[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-turboquant-vllm-comparison-fp8-kv-cache-zh":3,"article-related-turboquant-vllm-comparison-fp8-kv-cache-zh":31,"series-research-381fb6c6-6da7-4444-831f-8c5eed8d685c":85},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":23,"views":27,"created_at":28,"published_at":29,"topic_cluster_id":30},"381fb6c6-6da7-4444-831f-8c5eed8d685c","turboquant-vllm-comparison-fp8-kv-cache-zh","TurboQuant 與 FP8 實測結果","\u003Cp data-speakable=\"summary\">v\u003Ca href=\"\u002Fnews\u002Fllmbda-calculus-agent-safety-rules-zh\">LLM\u003C\u002Fa> 首次大規模比較 \u003Ca href=\"\u002Ftag\u002Fturboquant\">TurboQuant\u003C\u002Fa> 與 FP8 KV-cache。結果很直白：FP8 在速度上更穩，TurboQuant 的高壓縮版本則常掉準確率。\u003C\u002Fp>\u003Cp>\u003Ca href=\"https:\u002F\u002Fvllm.ai\u002Fblog\u002Fturboquant\" target=\"_blank\" rel=\"noopener\">vLLM\u003C\u002Fa> 在 2026 年 5 月 11 日發文。它把 \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.08671\" target=\"_blank\" rel=\"noopener\">TurboQuant\u003C\u002Fa> 拉進真實服務場景測。測了 4 個變體、4 個模型、5 個 \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa>。還同時比了 BF16 和 \u003Ca href=\"https:\u002F\u002Fvllm.ai\u002Fblog\u002Ffp8-kv-cache\" target=\"_blank\" rel=\"noopener\">FP8 KV-cache\u003C\u002Fa>。\u003C\u002Fp>\u003Cp>講白了，這不是小 demo。這是看伺服器真的扛不扛得住。KV-cache 一旦進到\u003Ca href=\"\u002Ftag\u002F長上下文\">長上下文\u003C\u002Fa>、高併發、記憶體吃緊的場景，速度和準確率就會一起露餡。\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>方法\u003C\u002Fth>\u003Cth>KV-cache 容量\u003C\u002Fth>\u003Cth>延遲影響\u003C\u002Fth>\u003Cth>吞吐影響\u003C\u002Fth>\u003Cth>準確率訊號\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>FP8\u003C\u002Ftd>\u003Ctd>2x\u003C\u002Ftd>\u003Ctd>幾乎沒有\u003C\u002Ftd>\u003Ctd>接近 BF16\u003C\u002Ftd>\u003Ctd>接近基準\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>TurboQuant k8v4\u003C\u002Ftd>\u003Ctd>2.4x\u003C\u002Ftd>\u003Ctd>慢 10% 到 68%\u003C\u002Ftd>\u003Ctd>BF16 的 80% 到 75%\u003C\u002Ftd>\u003Ctd>接近基準\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>TurboQuant 4bit-nc\u003C\u002Ftd>\u003Ctd>2.3x 到 3.7x\u003C\u002Ftd>\u003Ctd>有明顯變慢\u003C\u002Ftd>\u003Ctd>約 BF16 的 75%\u003C\u002Ftd>\u003Ctd>有中度下降\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>TurboQuant k3v4-nc \u002F 3bit-nc\u003C\u002Ftd>\u003Ctd>高於 FP8\u003C\u002Ftd>\u003Ctd>最慢\u003C\u002Ftd>\u003Ctd>BF16 的 66% 到 73%\u003C\u002Ftd>\u003Ctd>下降明顯\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>TurboQuant 為什麼會紅\u003C\u002Fh2>\u003Cp>TurboQuant 的做法很直接。它把 KV-cache 壓到 3 到 4 bit。之後再解量化回 BF16，才能做 attention。這跟 FP8 很不一樣。FP8 是直接存 FP8，attention 也能跑 FP8 Tensor \u003Ca href=\"\u002Fnews\u002Fnvidia-backs-corning-factories-with-billions-zh\">Cor\u003C\u002Fa>e。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778839867551-4v9g.png\" alt=\"TurboQuant 與 FP8 實測結果\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>差別就在這裡。TurboQuant 省記憶體很兇。可是在推理路徑裡，它多了一段解量化成本。你省下來的空間，常常又被額外運算吃回去。\u003C\u002Fp>\u003Cp>所以 vLLM 這篇比較像實測報告，不像產品宣傳。很多方法在簡報上很漂亮。可是一到延遲、吞吐、準確率，尤其是長提示詞和推理題，結果就很誠實。\u003C\u002Fp>\u003Cul>\u003Cli>測試的 TurboQuant 變體有 4 種：k8v4、4bit-nc、k3v4-nc、3bit-nc\u003C\u002Fli>\u003Cli>基準是 BF16 和 FP8\u003C\u002Fli>\u003Cli>模型包含 \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FMiniMaxAI\u002FMiniMax-M2.7\" target=\"_blank\" rel=\"noopener\">MiniMax-M2.7\u003C\u002Fa>、\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.3-70B-Instruct\" target=\"_blank\" rel=\"noopener\">Llama-3.3-70B-Instruct\u003C\u002Fa>、\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-30B-A3B-Instruct-2507\" target=\"_blank\" rel=\"noopener\">Qwen3-30B-A3B-Instruct-2507\u003C\u002Fa>、\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-30B-A3B-Thinking-2507\" target=\"_blank\" rel=\"noopener\">Qwen3-30B-A3B-Thinking-2507\u003C\u002Fa>\u003C\u002Fli>\u003Cli>benchmark 包含 \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenai\u002Fmrcr\" target=\"_blank\" rel=\"noopener\">openai\u002Fmrcr\u003C\u002Fa>、AIME25、GPQA:Diamond、MATH500、LiveCodeBench-v6\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>準確率到底掉多少\u003C\u002Fh2>\u003Cp>準確率的結論沒有那麼單一，但趨勢很清楚。FP8 和 TurboQuant k8v4 大多能貼近原始基準。這代表它們還算能用，至少不會一上線就把答案搞歪。\u003C\u002Fp>\u003Cp>4bit-nc 就開始有感了。它還在可討論範圍內。若你的伺服器真的卡在記憶體，這版本可能還能試。可是一旦壓到 k3v4-nc 或 3bit-nc，掉分就很明顯。\u003C\u002Fp>\u003Cp>在長上下文檢索上，Llama-3.3-70B-Instruct 於 128k context 的結果很有代表性。BF16 平均恢復率約 98%。4bit-nc 約 96%。k3v4-nc 和 3bit-nc 則大約少了 20 分。\u003C\u002Fp>\u003Cblockquote>\"FP8 via --kv-cache-dtype fp8 remains the best default for KV-cache quantization.\" — vLLM blog, 2026-05-11\u003C\u002Fblockquote>\u003Cp>這句話很直白。我也覺得很合理。你如果想保住準確率，又想省一點記憶體，FP8 就是最穩的預設值。TurboQuant 比較像特殊情境工具，不是通用答案。\u003C\u002Fp>\u003Cul>\u003Cli>長上下文檢索測到各模型支援的最長長度\u003C\u002Fli>\u003Cli>準確率是 5 次重複的平均 pass@1\u003C\u002Fli>\u003Cli>k3v4-nc 和 3bit-nc 在最難的長上下文案例大約掉 20 分\u003C\u002Fli>\u003Cli>MiniMax-M2.7 上，激進版本在 AIME25 與 LiveCodeBench-v6 最多掉約 8 分\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>速度這關，TurboQuant 輸得更明顯\u003C\u002Fh2>\u003Cp>速度結果更不漂亮。vLLM 用 1,024 個 input tokens 和 256 個 output tokens 來測延遲。batch size 也掃了 1、8、32、64。FP8 幾乎沒\u003Ca href=\"\u002Fnews\u002Fwhy-nebius-ai-pivot-is-more-real-than-hype-zh\">什麼\u003C\u002Fa>負擔。TurboQuant 就不是這樣。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778839865020-3hoj.png\" alt=\"TurboQuant 與 FP8 實測結果\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>在 Qwen3-30B-A3B-Instruct-2507 上，TurboQuant 的延遲開銷大約落在 10% 到 60%。在 Llama-3.3-70B-Instruct 上，範圍更大，約 10% 到 68%。而且 batch size 越大，開銷還會往上爬。這對服務團隊來說很煩。\u003C\u002Fp>\u003Cp>吞吐量也一樣。FP8 在兩個模型上都能貼近 BF16。TurboQuant 則低一截。Qwen3-30B 介於 BF16 的 80% 到 73%。Llama-3.3-70B 則是 75% 到 66%。\u003C\u002Fp>\u003Cp>這代表一件事。KV-cache 省下來的容量，不會自動變成更快的服務。你把解量化成本加回去，整個算式就變了。\u003C\u002Fp>\u003Cul>\u003Cli>延遲測試用了 10 次 warmup 和 30 次正式測試\u003C\u002Fli>\u003Cli>吞吐測試用了 200 個 prompts，token 組合為 256\u002F256、1024\u002F512、4096\u002F256\u003C\u002Fli>\u003Cli>vLLM 版本是 0.20.2，commit 為 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fcommit\u002F6ec9bbec3\" target=\"_blank\" rel=\"noopener\">6ec9bbec3\u003C\u002Fa>\u003C\u002Fli>\u003Cli>FP8 在延遲和吞吐上都接近 BF16，TurboQuant 則持續落後\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>對實際部署代表什麼\u003C\u002Fh2>\u003Cp>這份測試最實際的結論是，TurboQuant 是記憶體工具，不是效能工具。若你的模型服務真的卡在 KV-cache 容量，而且你能接受慢一點，那 TurboQuant 4bit-nc 可以先試。若你同時在意延遲、吞吐、準確率，FP8 比較乾脆。\u003C\u002Fp>\u003Cp>還有一個硬體面很重要。FP8 會吃到現代 \u003Ca href=\"\u002Ftag\u002Fnvidia\">NVIDIA\u003C\u002Fa> \u003Ca href=\"\u002Ftag\u002Fgpu\">GPU\u003C\u002Fa> 的原生 Tensor Core。TurboQuant 則要先把低 bit 資料拆開，attention 才能跑。慢的那一段，通常就卡在這裡。\u003C\u002Fp>\u003Cp>所以這篇文章對工程團隊的價值，不是給你一張漂亮圖表，而是幫你縮小實驗範圍。先試 FP8。只有在記憶體真的還不夠時，再去碰 TurboQuant。若你打算在 \u003Ca href=\"https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fdata-center\u002Fh100\u002F\" target=\"_blank\" rel=\"noopener\">H100\u003C\u002Fa> 上部署，問題就不是 TurboQuant 能不能省空間。問題是，你願不願意拿速度和準確率去換那點空間。\u003C\u002Fp>\u003Cp>我也建議把這篇和 OraCore 的 \u003Ca href=\"\u002Fnews\u002Ffp8-kv-cache-vllm\">FP8 KV-cache in vLLM\u003C\u002Fa>、\u003Ca href=\"\u002Fnews\u002Fkv-cache-optimization-guide\">KV-cache optimization strategies\u003C\u002Fa> 一起看。你會更快看懂，哪些優化是真的能上線，哪些只是實驗室裡好看。\u003C\u002Fp>\u003Ch2>結論\u003C\u002Fh2>\u003Cp>vLLM 這次的大規模比較很直接。TurboQuant 只有在記憶體壓力很大時才值得考慮。就算要用，FP8 還是大多數團隊該先試的預設值。\u003C\u002Fp>\u003Cp>如果你在做模型服務，我的建議很簡單。先量你的 KV-cache 壓力，再看延遲預算。若兩者都緊，就別急著追低 bit。先把 FP8 跑穩，才有資格談 TurboQuant。\u003C\u002Fp>","vLLM 首次大規模比較 TurboQuant 與 FP8 KV-cache。結果很直白：FP8 在速度上更穩，TurboQuant 的高壓縮版本則常掉準確率。","vllm.ai","https:\u002F\u002Fvllm.ai\u002Fblog\u002Fturboquant",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1778839867551-4v9g.png","research","zh","670a7f69-911f-41e8-a18b-7d3491253a19",[17,18,19,20,21,22],"TurboQuant","FP8 KV-cache","vLLM","KV-cache quantization","LLM 推理","模型服務",[24,25,26],"FP8 在 vLLM 的實測裡，比 TurboQuant 更穩，速度也更好。","TurboQuant 的低 bit 版本雖然省記憶體，但常付出延遲和準確率代價。","對多數部署來說，FP8 應該先試；TurboQuant 比較適合記憶體真的卡住的情境。",6,"2026-05-15T10:10:36.034569+00:00","2026-05-15T10:10:36.01+00:00","0c35a120-52fc-41fc-afa3-d404eb934158",{"tags":32,"relatedLang":44,"relatedPosts":48},[33,35,37,39,42],{"name":18,"slug":34},"fp8-kv-cache",{"name":19,"slug":36},"vllm",{"name":17,"slug":38},"turboquant",{"name":40,"slug":41},"KV cache quantization","kv-cache-quantization",{"name":21,"slug":43},"llm-推理",{"id":15,"slug":45,"title":46,"language":47},"turboquant-vllm-comparison-fp8-kv-cache-en","TurboQuant vs FP8: vLLM’s first broad test","en",[49,55,61,67,73,79],{"id":50,"slug":51,"title":52,"cover_image":53,"image_url":53,"created_at":54,"category":13},"33c9a55c-a8c0-4367-b742-f4567d1e98e3","mathematicians-warn-ai-could-distort-math-zh","數學界警告 AI 會扭曲證明標準","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780504386035-080l.png","2026-06-03T16:32:29.415063+00:00",{"id":56,"slug":57,"title":58,"cover_image":59,"image_url":59,"created_at":60,"category":13},"5c3cb90f-7efd-426f-8c09-32a303f82be9","humanoid-gpt-zero-shot-motion-tracking-zh","Humanoid-GPT：用 GPT 擴大動作追蹤","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780469319284-znpc.png","2026-06-03T06:47:34.463464+00:00",{"id":62,"slug":63,"title":64,"cover_image":65,"image_url":65,"created_at":66,"category":13},"e3a4b0f7-03b3-43c6-ae51-906b337c5c2f","ipt-vlms-hidden-space-reasoning-zh","IPT 讓 VLM 更會想像隱藏空間","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780468394735-1k40.png","2026-06-03T06:32:46.560029+00:00",{"id":68,"slug":69,"title":70,"cover_image":71,"image_url":71,"created_at":72,"category":13},"5fca9fe5-af66-47ce-85f0-0ffe1bee30b9","neuron-selectivity-changes-with-scale-zh","神經元選擇性會隨規模改變","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780467514422-7oss.png","2026-06-03T06:17:44.126547+00:00",{"id":74,"slug":75,"title":76,"cover_image":77,"image_url":77,"created_at":78,"category":13},"9f9c2a61-d058-4c62-bb88-106e683657f0","nasa-landsat-wild-disturbances-rising-zh","NASA Landsat：野火與風暴變多","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780448581102-owp0.png","2026-06-03T01:02:37.513233+00:00",{"id":80,"slug":81,"title":82,"cover_image":83,"image_url":83,"created_at":84,"category":13},"3479bdee-21fb-4fda-9572-9394caba01b0","adacodec-predictive-visual-code-video-mllms-zh","AdaCodec 用預測碼壓縮影片 token","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1780381988591-z2sp.png","2026-06-02T06:32:28.249023+00:00",[86,91,96,101,106,111,116,121,126,131],{"id":87,"slug":88,"title":89,"created_at":90},"f18dbadb-8c59-4723-84a4-6ad22746c77a","deepmind-bets-on-continuous-learning-ai-2026-zh","DeepMind 押注 2026 連續學習 AI","2026-03-26T08:16:02.367355+00:00",{"id":92,"slug":93,"title":94,"created_at":95},"f4a106cb-02a6-4508-8f39-9720a0a93cee","ml-papers-of-the-week-github-research-desk-zh","每週 ML 論文清單，為何紅到 GitHub","2026-03-27T01:11:39.284175+00:00",{"id":97,"slug":98,"title":99,"created_at":100},"c4f807ca-4e5f-47f1-a48c-961cf3fc44dc","ai-ml-conferences-to-watch-in-2026-zh","2026 AI 研討會投稿時程整理","2026-03-27T01:51:53.874432+00:00",{"id":102,"slug":103,"title":104,"created_at":105},"cf046742-efb2-4753-aef9-caed5da5e32e","adaptive-block-scaled-data-types-zh","IF4：神經網路量化的聰明選擇","2026-03-31T06:00:36.990273+00:00",{"id":107,"slug":108,"title":109,"created_at":110},"53a0dc54-0371-4e40-8d5e-74e94a73840c","geometry-aware-similarity-metrics-for-neural-representations-zh","超越距離測量：用微分幾何重新理解神經網路","2026-03-31T06:01:01.241968+00:00",{"id":112,"slug":113,"title":114,"created_at":115},"fee7d472-a775-4b1d-bbc2-1e8bca1bbf8b","on-the-fly-repulsion-in-the-contextual-space-for-rich-divers-zh","讓AI繪圖更有創意：用排斥力提升生成多樣性","2026-03-31T06:01:25.439673+00:00",{"id":117,"slug":118,"title":119,"created_at":120},"a9901203-d69b-447b-8854-15d14eab32b4","vision-aided-beam-prediction-cnn-eca-zh","影像輔助波束預測升級 CNN","2026-04-01T10:00:25.8073+00:00",{"id":122,"slug":123,"title":124,"created_at":125},"b55e7dd4-0a24-4b3d-804d-b0309a03f498","triple-band-fss-mimo-antenna-sub-6-ghz-zh","三頻 FSS MIMO 天線瞄準 sub-6 GHz","2026-04-01T13:18:36.857305+00:00",{"id":127,"slug":128,"title":129,"created_at":130},"f68290bd-e7f3-4b30-ba22-dcd4e0130a66","openclaw-1299-repos-eight-weeks-analysis-zh","OpenClaw 1299 個 Repo 的資料解讀","2026-04-02T05:03:45.208411+00:00",{"id":132,"slug":133,"title":134,"created_at":135},"ed9f80eb-eb02-4d35-8ad4-0ddf428751dd","beam-coherence-aware-combining-mmwave-mimo-zh","毫米波 MIMO 的雙階合併法","2026-04-02T05:27:26.897188+00:00"]