The Cost of Speaking a Mid-Resource Language to a Model

Two users send the same document to the same model. One document is in English, the other is its translation into Ukrainian. The content is identical. The second user pays roughly twice as much, receives a slower response, and hits the context limit sooner.

This is not a pricing policy. It is a consequence of how text is converted into the units a model operates on, and it has been measured precisely enough to be treated as an engineering parameter rather than a grievance.

What the measurements show

Tokenizer fertility — the average number of tokens a system produces per word — has been quantified across European languages. English sits at approximately 1.2 tokens per word. Ukrainian sits at approximately 2.7.

The distribution follows language families with some regularity. Romance languages cluster between 1.5 and 1.7. Germanic languages between 1.7 and 1.9. Slavic languages between 2.2 and 2.5. Uralic and Baltic languages between 2.7 and 3.0.

Ukrainian is the outlier within its own family. It costs 15 to 18 percent more than cognate Slavic languages — a gap the author attributes not to morphology, which Ukrainian shares with its neighbours, but to under-representation in pretraining data. The tokenizer learned to compress what it saw most of.

The ratios also vary substantially by model. On an identical corpus of Ukrainian legal text, fertility ranges across a factor of 1.6 between systems: from 2.43 tokens per word at the low end to 3.90 at the high end. For a thousand-word document that is a difference of roughly 1,468 tokens for the same input.

These figures are stable across domains. Fertility rankings correlate at above 0.97 between text types, which means the penalty is a property of the tokenizer and the language, not of the subject matter.

Ovcharov, “The Tokenizer Tax Across European Languages” (arXiv 2605.24718) and “Tokenizer Fertility and Zero-Shot Performance on Ukrainian Legal Text” (arXiv 2605.14890).

Why it compounds

Fragmentation is not a single cost. It appears in three places simultaneously.

Price. Commercial APIs bill per token. A user working in a high-fertility language pays more for identical content. Work presented at NeurIPS in 2023 established the general case: the same text translated across languages produces token counts differing by up to a factor of fifteen, and users of some languages pay at least 2.5 times more than English speakers for equivalent processing.

Context.A model's context window is measured in tokens, not meaning. A 128,000-token window holds substantially less Ukrainian than English. The limit binds earlier, and it binds on the same document.

Latency. More tokens means more forward passes. Generation takes longer and consumes more energy per unit of delivered content.

None of these is catastrophic in isolation. Together they mean that building anything in a mid-resource language — a public service, a legal research tool, an educational product — starts from a structural cost disadvantage that has nothing to do with the quality of the idea.

Petrov, La Malfa, Torr, Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023, arXiv 2305.15425).

The disadvantage has a source, and it is closing on its own terms

The representation gap in pretraining data is documented. In GPT-3, English accounted for 92.65 percent of the training corpus. French, the second-largest, accounted for 1.82 percent. Chinese accounted for 0.10 percent. Ukrainian did not appear as a separate line at all.

The figures for later models are less disclosed and probably less extreme, but the shape of the distribution has not fundamentally changed. Tokenizers built on such corpora encode the distribution they were built from.

What has changed is that closing the gap no longer requires training a model from scratch. Two things have made this true.

Corpora now exist at usable scale. The Kobza corpus assembles approximately 1.3 terabytes of Ukrainian text — nearly 60 billion tokens across 97 million documents — deduplicated from CulturaX, FineWeb 2, HPLT 2.0 and other sources. This is not a research curiosity; it is enough to pretrain on.

Tokenizer transfer works. Rather than accepting the tokenizer that ships with an open model, the vocabulary can be replaced and the embedding layer realigned. The Lapa project, built by researchers at the Ukrainian Catholic University, AGH University of Krakow and partner institutions, replaced 80,000 of 250,000 tokens in a Gemma-3-12B tokenizer with Ukrainian ones. The published result: the model requires 1.5 times fewer tokens for Ukrainian than the base model, and performs roughly three times fewer computations to reach better results.

That is not the elimination of the penalty. It is a substantial reduction of it, achieved without the capital required for a frontier training run.

Haltiuk and Smywiński-Pohl, “On the Path to Make Ukrainian a High-Resource Language” (UNLP 2025). Lapa model card, lapa-llm/lapa-v0.1.2-instruct.

Why this becomes more consequential, not less

Two developments make tokenizer efficiency a growing rather than shrinking concern.

The first is the projected exhaustion of human-generated public text. Epoch AI estimates the effectively available stock at approximately 300 trillion tokens, with an 80 percent confidence interval placing full utilisation somewhere between 2026 and 2032 — median around 2028. Overtraining, adopted to reduce inference costs, brings that date closer.

As the marginal value of additional English data falls, the relative value of well-curated corpora in under-represented languages rises. A language with a clean, deduplicated, licence-clear corpus becomes an asset rather than a rounding error.

The second is the constraint on synthetic substitution. Work published in Nature in 2024 established that models trained recursively on generated data degrade — rare distributional tails vanish first, and outputs converge toward distorted approximations. Subsequent work qualified this significantly: if synthetic data is accumulated alongside real data rather than replacing it, test error stays bounded rather than compounding across generations.

The qualification matters more than the headline. It means synthetic data is usable, but only against a base of genuine human text. Which returns the question to where that human text comes from, in which languages, and whether its provenance can be established — because a corpus contaminated by generated material is precisely what the accumulation strategy is supposed to avoid.

Villalobos et al., “Will we run out of data?” (Epoch AI). Shumailov et al., “AI models collapse when trained on recursively generated data”, Nature 631, 755–759 (2024). Gerstgrasser et al., arXiv 2404.01413.

What this argues for at the standards level

The tokenization penalty is unusual among problems in this field because it is fully measurable, uncontroversial in its mechanism, and already has demonstrated remedies.

That combination makes it a candidate for the kind of thing standards bodies handle well. Three specific items follow from the measurements above.

Disclosure of fertility by language. Model documentation reports parameter counts, context windows and benchmark scores. It does not report tokens-per-word by language. This is trivially measurable and directly determines the cost a user will incur. Its absence from model cards is an omission, not a technical obstacle.

Language coverage in evaluation. A benchmark suite run only in English establishes little about behaviour elsewhere. This holds for capability, and — as the refusal-rate measurements discussed in the previous analysis show — it holds for behaviour as well.

Corpus provenance as a documented property.If the value of human-generated text in under-represented languages is rising while the risk of synthetic contamination is also rising, then the ability to establish what a corpus contains stops being a compliance formality and becomes a question about the asset's worth.

The first two are documentation problems and could be addressed within existing frameworks. The third is not — it requires a technical mechanism that does not currently exist in any standardised form.