Token vs. word: why Chinese tokenization still matters

How do Chinese characters get split into tokens in modern LLMs?

This guide shows why Chinese tokenization can inflate token counts and how newer models reduce it.

Before you start

• An OpenAI account or API access for tokenization checks.
• A Qwen model endpoint or local Qwen-compatible tokenizer.
• Python 3.10+ or Node 20+ for quick experiments.
• Basic familiarity with tokens, tokenizers, and UTF-8 text.
• Access to official docs for OpenAI docs and the tiktoken GitHub repo.

Step 1: Inspect a Chinese string in a tokenizer

Goal: confirm that a short Chinese phrase can become multiple tokens, which is the core issue behind the “token vs. word” debate.

Use a tokenizer such as tiktoken to encode a few Chinese examples, including common characters and multi-character phrases.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
texts = ["淄博", "吃", "人工智能"]
for t in texts:
    ids = enc.encode(t)
    print(t, len(ids), ids)

You should see token counts greater than 1 for at least some Chinese inputs, which proves that token boundaries do not always match word boundaries.

Step 2: Compare GPT-2 style and newer vocabularies

Goal: understand why older tokenizers often split Chinese more aggressively than newer ones with larger vocabularies.

Run the same text through an older GPT-2 style tokenizer and then through a newer tokenizer used by current models. Compare the number of tokens for the same string.

You should see older vocabularies produce more fragments, while newer tokenizers often reduce the count for common Chinese characters.

Step 3: Test a model with Chinese-aware merges

Goal: verify how model-specific vocabularies can reduce token counts for frequent Chinese phrases.

Check a Qwen-compatible tokenizer with a phrase such as “人工智能” and compare it against a general-purpose tokenizer. Some vocabularies merge common Chinese phrases into a single token.

You should see fewer tokens for repeated or common Chinese phrases when the tokenizer was trained with Chinese-heavy merges.

Step 4: Estimate token cost for your prompts

Goal: translate tokenization behavior into cost, latency, and prompt budget impact.

Count tokens for the same Chinese prompt in multiple tokenizers, then multiply by your model’s input pricing or context limits. A prompt that looks short in characters can still consume a surprising number of tokens.

You should see that token count changes can affect both request cost and how much context remains for the rest of the conversation.

Step 5: Choose a tokenizer strategy for production

Goal: pick a practical rule for your app so Chinese input behaves predictably across models.

Use the tokenizer that matches the model you actually deploy, store token counts in tests, and add regression checks for common Chinese phrases your users send most often.

You should see stable token counts in your test suite, which helps prevent prompt-budget surprises after model upgrades.

Common mistakes

• Assuming one Chinese character always equals one token. Fix: measure with the exact tokenizer for your model.
• Using a tokenizer from a different model family. Fix: match the tokenizer to the deployed model, not to the language alone.
• Budgeting by character count instead of token count. Fix: calculate cost and context using token totals from real prompts.

What's next

Next, compare tokenization across OpenAI, Qwen, and another Chinese-focused model on your own corpus, then add those counts to prompt tests and cost monitoring.