How Tokenization Quietly Taxes Every AI Product You Ship
Tokenization is the subword-splitting step every large language model runs before it ever reasons about your prompt, and it isn't neutral. The same byte-pair-encoding process that makes English cheap to process can cost 50% to over 12x more tokens for other languages, and it explains why models stumble on letter-counting and arithmetic. As AI products expand into global markets, tokenizer behavior is turning into a hidden line item in unit economics and a real security surface, not just an engineering footnote. For PMs, the fix starts with actually measuring token counts across the languages and data types your product claims to support.
Do you actually know what your product costs to serve a user typing in Arabic, Thai, or Vietnamese — or have you only ever tested your token budget against English prompts? Most teams never check. Your eval set, cost model, and latency benchmarks were probably built on the language your team writes in, and that number quietly stops representing reality the moment a real international user opens the app. This week, do one thing: take your five most common prompt templates and run them through your model's actual tokenizer (OpenAI and Anthropic both publish tools for this) in the top three non-English languages your user base speaks. Compare token counts, not character counts. If the gap is anywhere near the 50-100% range research has documented for major world languages, your per-user cost model for that segment is wrong, and your context-window budget is tighter than English testing suggested. To be fair, you can't engineer this away entirely — you don't control the tokenizer, and switching vendors rarely closes the gap by much. But you can stop pricing and prompting blind. Bake language-specific token multipliers into your cost forecasts now, before finance asks why the "global" tier is bleeding margin you can't explain. The teams that get burned here find out from a P&L review instead of a spreadsheet they built themselves.
Byte-pair-encoding tokenizers used by GPT and Claude model families can require 50% to over 12x more tokens for non-English languages than English to express identical meaning, per multiple language-fairness studies.
Feed the word "hello" into GPT-5 and it costs you one token. Feed it the Thai greeting "สวัสดี" — same meaning, roughly the same length — and you can burn six or seven. Multiply that gap across every API call your product makes, and tokenization stops being invisible plumbing.
It's a line item.
Verdict: tokenization is the most underexamined cost and quality variable in production AI products, and it disproportionately punishes exactly the users — non-English speakers, structured data, precise numbers — that every "AI is going global" pitch deck promises to serve.
What's actually happening under the hood
Every major model family — OpenAI's GPT line, Anthropic's Claude line, Google's Gemini — breaks your text into subword chunks using some flavor of byte-pair encoding (BPE) before it ever reaches the neural network. OpenAI's current tokenizer, o200k_base, ships with a roughly 200,000-entry vocabulary shared across GPT-4o, the o-series, and GPT-5, according to OpenAI's published tiktoken library. Anthropic's Claude models use their own BPE-based vocabulary built on similar principles.
The algorithm is trained by starting with individual characters and repeatedly merging whichever adjacent pair shows up most often in the training corpus — "t"+"h" becomes "th," "th"+"e" becomes "the" — until it hits a target vocabulary size. Common English words end up as single tokens. Anything the training data saw rarely gets sawed into fragments.
Think of it like a vending machine stocked with exact-change coins for the currencies its designers use most. Feed it a familiar bill and it hands back one clean coin. Feed it an unfamiliar currency and it breaks your bill into a fistful of dimes and nickels just to represent the same value — more moving parts, more room for error, and a bigger bill.
To be fair to the model builders: nobody designed this to be unjust. BPE vocabularies get built by frequency-counting whatever training corpus is available, and the internet's training corpus is still disproportionately English. Fixing it means either training language-specific tokenizers, which fragments infrastructure, or growing shared vocabularies enormously, which slows every model down.
Neither is free.
The line item nobody puts in the pricing deck
Here's where it stops being academic. Research comparing tokenizer output across languages has found German and Italian text runs roughly 50% more expensive than equivalent English text on GPT-style tokenizers, while some low-resource languages — Dzongkha, Odia, Santali, and Shan among them — need more than 12 times as many tokens to express the same meaning. One widely cited comparison: at GPT-4o's published input rate of $2.50 per million tokens, translating the same content into Arabic ran about $4.93 per million words versus $2.90 for English — a 70% premium for saying the identical thing.
It isn't only OpenAI's tokenizer, either. A separate study measuring Claude's behavior on Thai text found it cost 2.22 times more to process than English — while scoring 5.1 percentage points lower on accuracy. You're paying more for a worse answer.
If your product's addressable market includes Bangkok, Riyadh, or Jakarta and your unit economics were modeled on an English-only eval set, your margins on international users are quietly worse than your dashboard shows, because your token-cost assumptions were never localized.
Why models can't spell or count
Tokenization also explains two of the most-mocked LLM failure modes, and it's worth knowing why so you stop expecting a fix from a bigger model. When you ask a model to count the letters in a word, it isn't looking at individual characters — it never saw them. It saw whatever subword chunks the tokenizer produced, and it has to reconstruct spelling from patterns learned during training, not direct observation.
That's the mechanical root of the infamous "how many r's in strawberry" stumble, and prompting tricks around it are workarounds, not fixes.
Arithmetic has a related problem. Digit sequences don't always tokenize into consistent place-value chunks — a four-digit number might arrive as one token in one context and split unevenly in another, depending on what the training data rewarded. The model isn't doing column addition the way you learned it in third grade; it's pattern-matching against token sequences it's seen before, which is exactly why it's more reliable on common numbers than obscure ones.
If your product does anything with precise counting, exact string manipulation, or multi-digit math, route that work to a code-execution tool. Don't ask the token stream to do arithmetic on your behalf.
The edge cases that bite in production
The subtler risk shows up in security. Content filters and prompt-injection defenses often pattern-match on raw text, but the model reasons over tokens — and the same string can tokenize differently depending on spacing, unicode variants, or invisible characters, sometimes slipping a flagged phrase past a filter while the model still "reads" it as intended. This is a documented class of tokenizer-boundary attack, not a hypothetical.
Anyone building guardrails on string matching alone is defending the wrong layer.
None of this means switch vendors on tokenizer efficiency alone — output quality and latency still dominate that decision. But if you're pricing a global product, evaluating a model swap, or debugging why non-English users churn faster, the honest question is whether your team has ever actually measured token counts per language and per data type, or whether you've just assumed the English benchmark generalizes. For most teams building "global from day one," it hasn't been measured at all.
That's the harder question worth putting on the next roadmap review: do you actually know what your product costs to run in the sixty languages your landing page claims to support?
Get this in your inbox. AI Rundown Daily delivers original briefings every morning — free. Subscribe →
Frequently Asked Questions
Partly because BPE vocabularies are built by frequency-counting the training corpus, and the open web is still disproportionately English, so fixing it means either training separate tokenizers per language family — fragmenting infrastructure and complicating fine-tuning — or dramatically expanding shared vocabulary size, which slows inference for everyone. Larger shared vocabularies, like OpenAI's o200k_base, have narrowed the gap somewhat for major world languages compared to older encodings. Low-resource languages remain far behind because there simply isn't enough training text to learn efficient chunks for them. This is a training-data problem wearing an engineering costume, and it won't fully resolve until non-English text is proportionally represented in training corpora.
Both OpenAI and Anthropic publish free tokenizer tools and libraries, like tiktoken and Anthropic's token-counting endpoint, that let you paste text and see the exact token count a given model will charge for. Run your actual production prompts and expected outputs through these tools in each target language, not synthetic test sentences, since real user phrasing tokenizes differently than clean text. Budget an afternoon, not a research sprint — the output is a multiplier table you can drop straight into your cost model.
The quiet failure is margin erosion: your per-user cost in a language you didn't test balloons past what your pricing tier assumes, and you don't notice until a finance review flags an unprofitable segment months later. The louder failure is a security one — a prompt-injection payload crafted to tokenize differently than it displays can slip past text-based content filters while the model still processes the intended malicious instruction. Neither failure announces itself with an error message; both show up as a bad quarter or a bad incident report.
