[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-atomicbot-llama-cpp-fork-throughput-gains-zh":3,"article-related-atomicbot-llama-cpp-fork-throughput-gains-zh":35,"series-industry-493ea70d-fffd-4365-ba76-63069ada5744":82},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":27,"views":31,"created_at":32,"published_at":33,"topic_cluster_id":34},"493ea70d-fffd-4365-ba76-63069ada5744","atomicbot-llama-cpp-fork-throughput-gains-zh","AtomicBot 的 llama.cpp 分支，兩條路都加速","\u003Cp data-speakable=\"summary\">這篇整理 AtomicBot 的 llama.cpp 分支如何同時從推理流程和記憶體壓縮兩端提速，幫你判斷 Gemma 4、Qwen 3.6、\u003Ca href=\"\u002Ftag\u002Fturboquant\">TurboQuant\u003C\u002Fa> KV 與權重壓縮哪個最值得先試。\u003C\u002Fp>\u003Cp>AtomicBot-ai 的 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FAtomicBot-ai\u002Fatomic-llama-cpp-turboquant\">atomic-llama-cpp-turboquant\u003C\u002Fa> 分支主打不換整套 serving 架構，也能把 tokens per second 往上推。依 repo 的 matrix bench，Gemma 4 的短提示吞吐最高可多 30-50%，TurboQuant 路徑則宣稱 KV 壓縮約 4.3 倍。讀完這 4 項，你大致就能判斷自己該先碰哪一條\u003Ca href=\"\u002Fnews\u002Fultraquant-4bit-kv-caching-agents-zh\">加速\u003C\u002Fa>線。\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>項目\u003C\u002Fth>\u003Cth>最佳場景\u003C\u002Fth>\u003Cth>報告增益\u003C\u002Fth>\u003Cth>主要限制\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>Gemma 4 MTP\u003C\u002Ftd>\u003Ctd>Gemma 4、頻寬受限\u003C\u002Ftd>\u003Ctd>約 30-50% 短提示吞吐\u003C\u002Ftd>\u003Ctd>需要 assistant head\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen 3.6 NextN\u003C\u002Ftd>\u003Ctd>Qwen 3.6 dense \u002F MoE\u003C\u002Ftd>\u003Ctd>35B-A3B 約 24-36%，27B dense 約 5-7%\u003C\u002Ftd>\u003Ctd>需 combined *_MTP.gguf\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>TurboQuant KV\u003C\u002Ftd>\u003Ctd>記憶體壓力大\u003C\u002Ftd>\u003Ctd>約 4.3 倍 KV 壓縮\u003C\u002Ftd>\u003Ctd>turbo3 設定效果較佳\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>TurboQuant 權重\u003C\u002Ftd>\u003Ctd>部署空間緊\u003C\u002Ftd>\u003Ctd>低位元權重壓縮\u003C\u002Ftd>\u003Ctd>依 backend 有取捨\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>1. Gemma 4 的 MTP 先把短提示拉快\u003C\u002Fh2>\u003Cp>這個分支最醒目的能力，是替 \u003Ca href=\"https:\u002F\u002Fai.google.dev\u002Fgemma\">Gemma 4\u003C\u002Fa> 加上 Multi-\u003Ca href=\"\u002Ftag\u002Ftoken\">Token\u003C\u002Fa> Prediction 的 speculative decoding。它透過 \u003Ccode>--mtp-head\u003C\u002Fcode> 載入官方 gemma4_assistant head，讓 draft 與驗證重疊執行，短提示時最容易看到吞吐提升。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782332275892-6iw2.png\" alt=\"AtomicBot 的 llama.cpp 分支，兩條路都加速\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>repo 的 matrix bench 顯示，Gemma 4 26B-A4B 與 31B 在 f16 KV 下，可拿到約 30-50% 的 throughput 增益。它也刻意避開常見 draft model 的額外成本，不需要第二個 context、第二個 tokenizer 或第二份 \u003Ca href=\"\u002Ftag\u002Fkv-cache\">KV cache\u003C\u002Fa>。\u003C\u002Fp>\u003Cul>\u003Cli>支援 Gemma 4 E2B、E4B、26B-A4B、31B\u003C\u002Fli>\u003Cli>建議 assistant quant：\u003Ccode>Q4_K_M\u003C\u002Fcode>\u003C\u002Fli>\u003Cli>非同步流程用 \u003Ccode>llama_decode_mtp_async\u003C\u002Fcode> 與 \u003Ccode>llama_decode_mtp_wait\u003C\u002Fcode>\u003C\u002Fli>\u003Cli>最適合頻寬先卡住、不是算力先卡住的工作負載\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>2. Qwen 3.6 的 NextN 走更直接的提速路線\u003C\u002Fh2>\u003Cp>如果你跑的是 \u003Ca href=\"https:\u002F\u002Fqwenlm.github.io\">Qwen\u003C\u002Fa>，這個分支提供 NextN speculative decoding，透過 \u003Ccode>--spec-type nextn\u003C\u002Fcode> 和 \u003Ccode>--model-draft\u003C\u002Fcode> 啟用。draft context 直接重用 target 的 \u003Ccode>llama_model\u003C\u002Fcode>，所以不必再掛第二次 mmap，部署也比獨立 assistant model 乾淨。\u003C\u002Fp>\u003Cp>repo 提到，Qwen 3.6 35B-A3B MoE 可提升約 24-36%，27B dense 在 MacBook Pro M4 Max 單槽測試下約 5-7%。如果你想要速度變快，但不想把 serving pipeline 改得太複雜，這條線很實用。\u003C\u002Fp>\u003Cul>\u003Cli>對應 Qwen 3.6 27B dense 與 35B-A3B MoE\u003C\u002Fli>\u003Cli>使用合併後的 \u003Ccode>*_MTP.gguf\u003C\u002Fcode> draft\u003C\u002Fli>\u003Cli>可搭配 AtomicChat Qwen 3.6 UDT collection\u003C\u002Fli>\u003Cli>draft tensor 固定在 \u003Ccode>Q8_0\u003C\u002Fcode>，提升接受穩定性\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>3. TurboQuant KV 壓縮，先把記憶體壓力降下來\u003C\u002Fh2>\u003Cp>TurboQuant 是這個分支的另一條主線，重點放在 KV cache 壓縮。它採用 WHT 旋轉的低位元量化，並為 Metal TurboFlash、\u003Ca href=\"\u002Ftag\u002Fcuda\">CUDA\u003C\u002Fa>、Vulkan、HIP 提供 backend 原生 kernels。當 context 變長或 batch 壓力升高時，這種做法特別有感。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782332276121-xa92.png\" alt=\"AtomicBot 的 llama.cpp 分支，兩條路都加速\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>專案宣稱 \u003Ccode>-ctk turbo3 -ctv turbo3\u003C\u002Fcode> 可達約 4.3 倍 KV 壓縮。對 memory-bound 的\u003Ca href=\"\u002Fnews\u002Fdatabricks-model-serving-supported-models-regions-zh\">模型\u003C\u002Fa>來說，這通常比單純追求算力更有效，因為它直接減少了記憶體流量與 device 端\u003Ca href=\"\u002Fnews\u002F10-ai-agent-workflows-b2b-catalog-leads-zh\">工作\u003C\u002Fa>集。\u003C\u002Fp>\u003Ccode>-ctk turbo3 -ctv turbo3\n--draft-block-size 3\n-ngl 99 -ngld 99\u003C\u002Fcode>\u003Ch2>4. TurboQuant 權重壓縮，讓模型先瘦身再上線\u003C\u002Fh2>\u003Cp>除了 KV cache，這個分支也支援低位元權重壓縮，像 \u003Ccode>TQ4_1S\u003C\u002Fcode>、\u003Ccode>TQ3_1S\u003C\u002Fcode> 這類格式。這讓你在 \u003Ca href=\"\u002Ftag\u002Finference\">inference\u003C\u002Fa> 開始前就先把 footprint 壓小，對筆電、小型 GPU，或 CPU-GPU 混合部署都很實際。\u003C\u002Fp>\u003Cp>它不只是省硬碟空間。更小的權重通常代表較快載入、更低記憶體壓力，也更容易和既有 GGUF 流程搭配。如果你本來就熟悉 llama.cpp 的模型格式，這一項最容易直接試跑。\u003C\u002Fp>\u003Cul>\u003Cli>常見格式：\u003Ccode>TQ4_1S\u003C\u002Fcode>、\u003Ccode>TQ3_1S\u003C\u002Fcode>\u003C\u002Fli>\u003Cli>適合模型大小本身就是瓶頸的場景\u003C\u002Fli>\u003Cli>可和量化 assistant head 一起使用\u003C\u002Fli>\u003Cli>仍沿用同一套 llama.cpp serving 流程\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>5. 多模態與快取整理，讓混合工作流少一點摩擦\u003C\u002Fh2>\u003Cp>這個分支也把 speculative decoding 延伸到多模態服務。README 提到 \u003Ccode>--mmproj\u003C\u002Fcode> 可以和 MTP、NextN 或 Eagle3 一起在單槽載入，文字輪次享受 draft 加速，圖片輪次則回到一般 target decoding。\u003C\u002Fp>\u003Cp>另一個實用細節是 \u003Ccode>-hf\u003C\u002Fcode> 下載的 Hugging Face cache 位置改成標準路徑，方便和其他工具共用，也比較不會在不同環境之間留下難整理的模型檔。\u003C\u002Fp>\u003Cul>\u003Cli>單槽支援多模態加 speculative decoding\u003C\u002Fli>\u003Cli>文字輪次可走 draft acceleration\u003C\u002Fli>\u003Cli>圖片輪次維持 target decoding\u003C\u002Fli>\u003Cli>\u003Ca href=\"https:\u002F\u002Fhuggingface.co\">Hugging Face\u003C\u002Fa> cache 位置更貼近標準工具習慣\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>怎麼挑\u003C\u002Fh2>\u003Cp>如果你跑 Gemma 4，而且瓶頸是記憶體頻寬，先試 MTP，再看 TurboQuant KV。若你主要是 Qwen 3.6，用 NextN 會更直接，尤其是 35B-A3B MoE，repo 報告的提升最明顯。兩種情況都一樣，這個分支的價值在於不離開 llama.cpp，就能先拿到一段可觀的加速。\u003C\u002Fp>\u003Cp>如果你最在意的是縮小記憶體占用，先從 TurboQuant KV 和權重壓縮下手。若你的工作負載以文字為主、又特別在意短提示延遲，MTP 會是最值得先試的一項。若你同時處理圖文混合流量，多模態路徑可以看，但圖片輪次仍會比較像一般 target decoding。\u003C\u002Fp>","4 項改動看懂 AtomicBot 的 llama.cpp 分支：Gemma 4、Qwen 3.6、TurboQuant KV 與權重壓縮，最快可達 30-50% 吞吐提升。","github.com","https:\u002F\u002Fgithub.com\u002FAtomicBot-ai\u002Fatomic-llama-cpp-turboquant",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782332275892-6iw2.png","industry","zh","cc87056f-b2e8-4ef0-966c-bf82ccffbb54",[17,18,19,20,21,22,23,24,25,26],"llama.cpp","AtomicBot","Gemma 4","Qwen 3.6","MTP","NextN","TurboQuant","KV cache compression","GGUF","speculative decoding",[28,29,30],"Gemma 4 用 MTP 最容易拿到 30-50% 短提示吞吐提升。","Qwen 3.6 的 NextN 更適合不想改太多 serving 架構的用戶。","TurboQuant KV 可把記憶體壓力明顯降下來，宣稱約 4.3 倍壓縮。",0,"2026-06-24T20:17:28.725554+00:00","2026-06-24T20:17:28.719+00:00","caa87b65-9bbc-46fe-bba8-4f4158dd2d8b",{"tags":36,"relatedLang":41,"relatedPosts":45},[37,39],{"name":19,"slug":38},"gemma-4",{"name":17,"slug":40},"llamacpp",{"id":15,"slug":42,"title":43,"language":44},"atomicbot-llama-cpp-fork-throughput-gains-en","AtomicBot’s llama.cpp fork boosts throughput on two fronts","en",[46,52,58,64,70,76],{"id":47,"slug":48,"title":49,"cover_image":50,"image_url":50,"created_at":51,"category":13},"707054b6-d7b6-46c2-89f8-161bb4e6f37c","chen-liwu-intel-packaging-materials-podcast-zh","陈立武把英特尔改成材料公司","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782342200185-fjma.png","2026-06-24T23:02:57.434146+00:00",{"id":53,"slug":54,"title":55,"cover_image":56,"image_url":56,"created_at":57,"category":13},"a7139e35-f8f1-498d-84d8-5a3c1c7c0192","zilliz-vector-lakebase-unified-ai-data-platform-zh","Zilliz 把向量搜尋收進單一 AI 資料層","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782339470762-t7nh.png","2026-06-24T22:17:21.056184+00:00",{"id":59,"slug":60,"title":61,"cover_image":62,"image_url":62,"created_at":63,"category":13},"c97ce8e9-7e93-488d-b11c-c1f54d71353c","nvidia-ceo-ai-lift-software-stocks-zh","Nvidia CEO：AI 反而能撐軟體股","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782328675330-1usd.png","2026-06-24T19:17:27.597152+00:00",{"id":65,"slug":66,"title":67,"cover_image":68,"image_url":68,"created_at":69,"category":13},"0ca173ef-42e8-413e-9631-b68375b5793b","openai-statistics-2026-users-revenue-funding-zh","OpenAI 2026 數據總覽","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782321481216-sj89.png","2026-06-24T17:17:35.945196+00:00",{"id":71,"slug":72,"title":73,"cover_image":74,"image_url":74,"created_at":75,"category":13},"fba9c922-207c-4e55-b54e-d4ea5e91af80","openai-custom-chip-right-move-against-nvidia-zh","OpenAI 自研晶片是對抗 Nvidia 的正確一步","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782320566922-cfk8.png","2026-06-24T17:02:18.227111+00:00",{"id":77,"slug":78,"title":79,"cover_image":80,"image_url":80,"created_at":81,"category":13},"b5835d28-30fa-4249-bf46-eb9f99ac114b","10-ai-agent-workflows-b2b-catalog-leads-zh","10 個把 B2B 型錄變成線索的 AI 工作流","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782313388636-gmuf.png","2026-06-24T15:02:32.787538+00:00",[83,88,93,98,103,108,113,118,123,128],{"id":84,"slug":85,"title":86,"created_at":87},"ee073da7-28b3-4752-a319-5a501459fb87","ai-in-2026-what-actually-matters-now-zh","2026 AI 真正重要的事","2026-03-26T07:09:12.008134+00:00",{"id":89,"slug":90,"title":91,"created_at":92},"83bd1795-8548-44c9-9a7e-de50a0923f71","trump-ai-framework-power-speech-state-preemption-zh","川普 AI 框架瞄準電力、言論與州權","2026-03-26T07:12:18.695466+00:00",{"id":94,"slug":95,"title":96,"created_at":97},"ea6be18b-c903-4e54-97b7-5f7447a612e0","nvidia-gtc-2026-big-ai-announcements-zh","NVIDIA GTC 2026 重點拆解","2026-03-26T07:14:26.62638+00:00",{"id":99,"slug":100,"title":101,"created_at":102},"4bcec76f-4c36-4daa-909f-54cd702f7c93","claude-users-spreading-out-and-getting-better-zh","Claude 用戶更分散，也更會用","2026-03-26T07:22:52.325888+00:00",{"id":104,"slug":105,"title":106,"created_at":107},"bd903b15-2473-4178-9789-b7557816e535","openclaw-raises-hard-question-for-ai-models-zh","OpenClaw 逼問 AI 模型價值","2026-03-26T07:24:54.707486+00:00",{"id":109,"slug":110,"title":111,"created_at":112},"eeac6b9e-ad9d-4831-8eec-8bba3f9bca6a","gap-google-gemini-checkout-fashion-search-zh","Gap 把結帳搬進 Gemini","2026-03-26T07:28:23.937768+00:00",{"id":114,"slug":115,"title":116,"created_at":117},"0740e53f-605d-4d57-8601-c10beb126f3c","google-pushes-gemini-transition-to-march-2026-zh","Google 把 Gemini 轉換延到 2026 年 3…","2026-03-26T07:30:12.825269+00:00",{"id":119,"slug":120,"title":121,"created_at":122},"e660d801-2421-4529-8fa9-86b82b066990","metas-llama-4-benchmark-scandal-gets-worse-zh","Meta Llama 4 分數風波又擴大","2026-03-26T07:34:21.156421+00:00",{"id":124,"slug":125,"title":126,"created_at":127},"183f9e7c-e143-40bb-a6d5-67ba84a3a8bc","accenture-mistral-ai-sovereign-enterprise-deal-zh","Accenture 攜手 Mistral AI 賣主權 AI","2026-03-26T07:38:14.818906+00:00",{"id":129,"slug":130,"title":131,"created_at":132},"191d9b1b-768a-478c-978c-dd7431a38149","mistral-ai-faces-its-hardest-year-yet-zh","Mistral AI 迎來最硬的一年","2026-03-26T07:40:23.716374+00:00"]