Реклама

SuTRA : Structurally-Unified Tokenization with Root Awareness

#SuTRA #Structurally-Unified #Tokenization #Root #Awareness

arXiv:2608.18087v1 Announce Type: new Abstract: Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex…