[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-token-ciyuan-zhongwen-fenciqi-shice-zh":3,"article-related-token-ciyuan-zhongwen-fenciqi-shice-zh":29,"series-tools-1b9f3648-fe4a-445d-bdc0-fda420e42c6b":74},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":11},"1b9f3648-fe4a-445d-bdc0-fda420e42c6b","token-ciyuan-zhongwen-fenciqi-shice-zh","用詞元實測中文分詞器","\u003Cp>想弄清楚大模型裡的 \u003Ca href=\"\u002Ftag\u002Ftoken\">Token\u003C\u002Fa>、詞元和中文分詞到底差在哪嗎？\u003C\u002Fp>\u003Cp data-speakable=\"summary\">這篇教你用同一組中文樣例實測不同 tokenizer，對照詞元數、切分方式與模型差異。\u003C\u002Fp>\u003Cp>你會搭好一個最小可重現環境，跑同一組中文樣例，看到不同模型對漢字、詞組和 UTF-8 字節的切分差異，並學會用結果判斷「一個字幾個 token」。\u003C\u002Fp>\u003Ch2>開始之前\u003C\u002Fh2>\u003Cul>\u003Cli>Node 20+\u003C\u002Fli>\u003Cli>Python 3.10+\u003C\u002Fli>\u003Cli>OpenAI API key，若要比較 OpenAI 模型\u003C\u002Fli>\u003Cli>Qwen 模型存取權或本地 tokenizer 檔，若要比較 Qwen\u003C\u002Fli>\u003Cli>Git\u003C\u002Fli>\u003Cli>可連網讀取 \u003Ca href=\"https:\u002F\u002Fplatform.openai.com\u002Fdocs\" target=\"_blank\" rel=\"noreferrer\">OpenAI Docs\u003C\u002Fa> 與 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fopenai\u002Ftiktoken\" target=\"_blank\" rel=\"noreferrer\">openai\u002Ftiktoken\u003C\u002Fa>\u003C\u002Fli>\u003C\u002Ful>\u003Cp>\u003Ca href=\"\u002Fnews\u002Fprepare-for-gemini-3-5-pro-on-launch-day-zh\">準備\u003C\u002Fa>好這些後，你就能直接復現實驗，而不是只看別人轉述「中文稅」或「一個漢字幾個 token」。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786320173760-2fu6.png\" alt=\"用詞元實測中文分詞器\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Ch2>Step 1: 建立詞元測試資料夾\u003C\u002Fh2>\u003Cp>目的：先做出一個乾淨工作區，專門放中文切分測試檔案與腳本。\u003C\u002Fp>\u003Cpre>\u003Ccode>mkdir tokenizer-check && cd tokenizer-check\npython -m venv .venv\nsource .venv\u002Fbin\u002Factivate\npip install tiktoken\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該看到新的專案資料夾、已啟用的虛擬環境，以及 \u003Ccode>tiktoken\u003C\u002Fcode> 安裝成功。\u003C\u002Fp>\u003Ch2>Step 2: 準備中文樣例清單\u003C\u002Fh2>\u003Cp>目的：建立一組同時包含單字、常用詞與多字詞組的測試集。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786320169650-mdki.png\" alt=\"用詞元實測中文分詞器\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cpre>\u003Ccode>cat > samples.txt << 'EOF'\n吃\n淄\n博\n人工智能\nToken\n詞元\nEOF\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該看到 \u003Ccode>samples.txt\u003C\u002Fcode> 有六行，包含單一漢字與四字詞組「人工智能」。\u003C\u002Fp>\u003Ch2>Step 3: 量出 OpenAI 詞元數\u003C\u002Fh2>\u003Cp>目的：測量 \u003Ca href=\"\u002Ftag\u002Fopenai\">OpenAI\u003C\u002Fa> 風格 tokenizer 如何切分每個樣例，方便你對照字級與詞組級行為。\u003C\u002Fp>\u003Cpre>\u003Ccode>python - << 'PY'\nimport tiktoken\nenc = tiktoken.get_encoding(\"cl100k_base\")\nfor s in [\"吃\", \"淄\", \"博\", \"人工智能\", \"Token\", \"詞元\"]:\n    ids = enc.encode(s)\n    print(f\"{s}\\t{len(ids)}\\t{ids}\")\nPY\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該看到每個樣例旁邊都印出 token 數與 id，且部分中文可能會出現多於 1 個 token。\u003C\u002Fp>\u003Ch2>Step 4: 比較不同詞表版本\u003C\u002Fh2>\u003Cp>目的：檢查較新的詞表是否把常見中文合併成更少的詞元。\u003C\u002Fp>\u003Cpre>\u003Ccode>python - << 'PY'\nimport tiktoken\nsamples = [\"吃\", \"淄\", \"博\", \"人工智能\"]\nfor name in [\"cl100k_base\", \"o200k_base\"]:\n    enc = tiktoken.get_encoding(name)\n    print(\"==\", name)\n    for s in samples:\n        print(s, len(enc.encode(s)))\nPY\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該看到兩個 tokenizer 名稱，並且至少有一個中文樣例的數量不同，這代表詞表版本會改變切分結果。\u003C\u002Fp>\u003Ch2>Step 5: 測試 Qwen tokenizer\u003C\u002Fh2>\u003Cp>目的：確認 \u003Ca href=\"\u002Fnews\u002Fqwen-3-8-max-cli-integration-protocol-migration-zh\">Qwen\u003C\u002Fa> tokenizer 是否把「人工智能」這類詞組合併\u003Ca href=\"\u002Fnews\u002Fnvidia-investor-page-disclosure-hub-zh\">成單一\u003C\u002Fa>詞元。\u003C\u002Fp>\u003Cpre>\u003Ccode>python - << 'PY'\nfrom transformers import AutoTokenizer\n\ntok = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-7B-Instruct\")\nfor s in [\"吃\", \"淄\", \"博\", \"人工智能\"]:\n    ids = tok.encode(s, add_special_tokens=False)\n    print(f\"{s}\\t{len(ids)}\\t{ids}\")\nPY\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該看到詞組與單字的計數結果，若「人工智能」顯示為 1，就代表它被合併成單一詞元。\u003C\u002Fp>\u003Ch2>Step 6: 整理比較表\u003C\u002Fh2>\u003Cp>目的：把原始計數整理成可直接引用的結果表，方便報告或分享。\u003C\u002Fp>\u003Cpre>\u003Ccode>python - << 'PY'\nresults = {\n    \"吃\": {\"OpenAI\": 2, \"Qwen\": 1},\n    \"淄\": {\"OpenAI\": 2, \"Qwen\": 1},\n    \"博\": {\"OpenAI\": 2, \"Qwen\": 1},\n    \"人工智能\": {\"OpenAI\": 4, \"Qwen\": 1},\n}\nfor s, row in results.items():\n    print(s, row)\nPY\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>驗收：你應該得到一份簡潔對照，能一眼看出詞組合併與逐字切分的差異。\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>指標\u003C\u002Fth>\u003Cth>基準／優化前\u003C\u002Fth>\u003Cth>結果／優化後\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>中文單字切分\u003C\u002Ftd>\u003Ctd>較舊的字節型 tokenizer 可能讓每個漢字需要 2 到 3 個詞元\u003C\u002Ftd>\u003Ctd>較新的詞表常把常見漢字降到 1 個詞元\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>詞組合併\u003C\u002Ftd>\u003Ctd>「人工智能」可能被拆成多個詞元\u003C\u002Ftd>\u003Ctd>某些 tokenizer 會把它合併成 1 個詞元\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>實務判斷\u003C\u002Ftd>\u003Ctd>詞元數不等於字數\u003C\u002Ftd>\u003Ctd>詞元數必須依模型逐一量測\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>常見錯誤\u003C\u002Fh2>\u003Cul>\u003Cli>只測一個 tokenizer 就推論所有模型。修法：每個目標模型都要各自測一次。\u003C\u002Fli>\u003Cli>把詞、字、詞元混為一談。修法：記住詞元可能是字節片段、單字或合併詞組。\u003C\u002Fli>\u003Cli>忽略標點與英文混排。修法：把 \u003Ccode>Token\u003C\u002Fcode>、數字與標點一起放進測試集。\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>接下來可以看什麼\u003C\u002Fh2>\u003Cp>當你能穩定量出詞元數後，下一步可以接著做提示詞成本估算、比較中英文效率，並判斷某個模型的 tokenizer 是否適合你的生產工作負載。\u003C\u002Fp>","這篇教你用同一組中文樣例實測不同 tokenizer，對照詞元數、切分方式與模型差異。","www.zhihu.com","https:\u002F\u002Fwww.zhihu.com\u002Fquestion\u002F2068659657892278726\u002Fanswer\u002F2068825611779543307",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786320173760-2fu6.png","tools","zh","0d0d262b-65b7-4388-82e5-bab51244f9c0",[17,18,19,20,21],"tiktoken","tokenizer","中文分詞","Qwen","OpenAI API",[23,24,25],"用同一組中文樣例比較不同 tokenizer，才能看出切分差異。","詞元數不是字數，必須依模型與詞表版本實測。","把結果整理成表格，最適合用來做模型選型與成本估算。",1,"2026-08-10T00:02:31.714597+00:00","2026-08-10T00:02:31.705+00:00",{"tags":30,"relatedLang":33,"relatedPosts":37},[31],{"name":20,"slug":32},"qwen",{"id":15,"slug":34,"title":35,"language":36},"token-vs-word-chinese-tokenization-matters-en","Token vs. word: why Chinese tokenization still matters","en",[38,44,50,56,62,68],{"id":39,"slug":40,"title":41,"cover_image":42,"image_url":42,"created_at":43,"category":13},"fc9bd7b8-542f-4c91-9965-60ecb3ae417e","deepseek-codex-ai-coding-costs-reset-zh","DeepSeek 接入 Codex 後，AI 編程成本必須重算","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786321973140-5qz2.png","2026-08-10T00:32:32.710718+00:00",{"id":45,"slug":46,"title":47,"cover_image":48,"image_url":48,"created_at":49,"category":13},"0afe05ab-6675-498f-b179-531a8460290b","openai-api-pricing-august-2026-token-costs-zh","OpenAI API 單價落差到 180 美元","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786300368036-5pzh.png","2026-08-09T18:32:26.525684+00:00",{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"e9622f39-b7db-4d48-bea9-84bc671402cc","usage-limits-chatgpt-enterprise-edu-controls-zh","ChatGPT Enterprise 與 Edu 的用量上限，必須是核心管理控制","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786298570504-ubv3.png","2026-08-09T18:02:22.954575+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"b13d8b40-53c1-4249-a342-b0f40834ba06","prepare-for-gemini-3-5-pro-on-launch-day-zh","Gemini 3.5 Pro 上線日準備清單","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786276970495-cnpl.png","2026-08-09T12:02:25.680556+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"01ba2b4b-f67e-4fdf-8b32-85fcc989549c","kitesurf-turns-workers-into-agent-browser-zh","Kitesurf 把 Workers 變成代理瀏覽器","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786127603865-k5ht.png","2026-08-07T18:32:56.343308+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"def73103-111d-4f21-8eee-653cc18e9f04","cuda-warps-memory-divergence-explained-zh","CUDA 讓 warp 變成一台機器","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786064608556-bymw.png","2026-08-07T01:02:59.304763+00:00",[75,80,85,90,95,100,105,110,115,120],{"id":76,"slug":77,"title":78,"created_at":79},"855cd52f-6fab-46cc-a7c1-42195e8a0de4","surepath-real-time-mcp-policy-controls-zh","SurePath 推出即時 MCP 政策控管","2026-03-26T07:57:40.77233+00:00",{"id":81,"slug":82,"title":83,"created_at":84},"9b19ab54-edef-4dbd-9ce4-a51e4bae4ebb","mcp-in-2026-the-ai-tool-layer-teams-use-zh","2026 年 MCP：團隊真的在用的 AI 工具層","2026-03-26T08:01:46.589694+00:00",{"id":86,"slug":87,"title":88,"created_at":89},"af9c46c3-7a28-410b-9f04-32b3de30a68c","prompting-in-2026-what-actually-works-zh","2026 提示工程，真正有用的是什麼","2026-03-26T08:08:12.453028+00:00",{"id":91,"slug":92,"title":93,"created_at":94},"05553086-6ed0-4758-81fd-6cab24b575e0","garry-tan-open-sources-claude-code-toolkit-zh","Garry Tan 開源 Claude Code 工具包","2026-03-26T08:26:20.068737+00:00",{"id":96,"slug":97,"title":98,"created_at":99},"042a73a2-18a2-433d-9e8f-9802b9559aac","github-ai-projects-to-watch-in-2026-zh","2026 必看 20 個 GitHub AI 專案","2026-03-26T08:28:09.619964+00:00",{"id":101,"slug":102,"title":103,"created_at":104},"a5f94120-ac0d-4483-9a8b-63590071ac6a","claude-code-vs-cursor-2026-zh","Claude Code 與 Cursor 深度對比：202…","2026-03-26T13:27:14.279193+00:00",{"id":106,"slug":107,"title":108,"created_at":109},"0975afa1-e0c7-4130-a20d-d890eaed995e","practical-github-guide-learning-ml-2026-zh","2026 機器學習入門 GitHub 實用指南","2026-03-27T01:16:49.712576+00:00",{"id":111,"slug":112,"title":113,"created_at":114},"bfdb467a-290f-4a80-b3a9-6f081afb6dff","aiml-2026-student-ai-ml-lab-repo-review-zh","AIML-2026：像課綱的學生實驗 Repo","2026-03-27T01:21:51.467798+00:00",{"id":116,"slug":117,"title":118,"created_at":119},"80cabc3e-09fc-4ff5-8f07-b8d68f5ae545","ai-trending-github-repos-and-research-feeds-zh","AI Trending：把 AI 資源收成一張表","2026-03-27T01:31:35.262183+00:00",{"id":121,"slug":122,"title":123,"created_at":124},"3ce6e6e2-bac5-463e-9f8d-45caabcc61f7","awesome-ai-for-science-research-tools-map-zh","AI 科研工具清單，開始像地圖了","2026-03-27T01:46:50.521945+00:00"]