[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-turboquant-amd-gpus-kv-cache-latency-zh":3,"article-related-turboquant-amd-gpus-kv-cache-latency-zh":33,"series-industry-4fae6813-4bb1-459e-9556-1bd8b0b4ca4e":87},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":25,"views":29,"created_at":30,"published_at":31,"topic_cluster_id":32},"4fae6813-4bb1-459e-9556-1bd8b0b4ca4e","turboquant-amd-gpus-kv-cache-latency-zh","TurboQuant 在 AMD GPU 上把長上下文延遲壓下來","\u003Cp data-speakable=\"summary\">這篇整理 \u003Ca href=\"\u002Ftag\u002Fturboquant\">TurboQuant\u003C\u002Fa> 在 AMD GPU 上的 5 個實作選擇，幫你判斷\u003Ca href=\"\u002Ftag\u002F長上下文\">長上下文\u003C\u002Fa>服務該怎麼壓低 KV-cache 延遲。\u003C\u002Fp>\u003Cp>如果你的 \u003Ca href=\"\u002Ftag\u002Fllm\">LLM\u003C\u002Fa> 服務常被 KV-cache 逼到記憶體瓶頸，這份清單可以直接幫你決定要不要上 TurboQuant、選哪種量化設定，以及哪些技巧值得先做。ROCm 版本在實測中把端到端效能拉到最高 3.6x，TTFT 也從 13.9 秒降到 0.89 秒。\u003C\u002Fp>\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>項目\u003C\u002Fth>\u003Cth>規格 A\u003C\u002Fth>\u003Cth>規格 B\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd>TQ4\u002F4\u003C\u002Ftd>\u003Ctd>4-bit K \u002F 4-bit V\u003C\u002Ftd>\u003Ctd>預設推薦\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Agentic workload test\u003C\u002Ftd>\u003Ctd>100 conversations, 32 concurrency\u003C\u002Ftd>\u003Ctd>~25K prefixes\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Cache hit rate\u003C\u002Ftd>\u003Ctd>FP8\u003C\u002Ftd>\u003Ctd>5.3%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Cache hit rate\u003C\u002Ftd>\u003Ctd>TQ4\u002F4\u003C\u002Ftd>\u003Ctd>67.7%\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>End-to-end speedup\u003C\u002Ftd>\u003Ctd>ROCm kernels vs open-source baseline\u003C\u002Ftd>\u003Ctd>Up to 3.6x\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\u003Ch2>1. 先把 TurboQuant 做成可上線版本\u003C\u002Fh2>\u003Cp>這篇最重要的\u003Ca href=\"\u002Fnews\u002Fgoogles-monthly-ai-recaps-are-strategy-not-scrapbook-zh\">訊號\u003C\u002Fa>，不是 TurboQuant 會壓縮 KV-cache，而是 AMD 的 \u003Ca href=\"https:\u002F\u002Frocm.docs.amd.com\u002F\">ROCm\u003C\u002Fa> 實作把它做到了可服務化。對線上推理來說，Kernel 品質、記憶體行為和延遲表現，往往比演算法名稱更關鍵。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781299071883-1324.png\" alt=\"TurboQuant 在 AMD GPU 上把長上下文延遲壓下來\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>作者把版本整合進 \u003Ca href=\"https:\u002F\u002Fdocs.vllm.ai\u002F\">vLLM\u003C\u002Fa>，並用 Triton、HIP 和 FlyDSL 針對執行\u003Ca href=\"\u002Fnews\u002Faws-kubernetes-managed-clusters-path-zh\">路徑\u003C\u002Fa>做優化。這代表它不是只在論文圖表裡好看，而是朝著真實 serving 場景去修正瓶頸。\u003C\u002Fp>\u003Cul>\u003Cli>目標平台：AMD Instinct GPU 上的 vLLM\u003C\u002Fli>\u003Cli>優化組合：Triton、HIP ISA 控制、FlyDSL\u003C\u002Fli>\u003Cli>核心目標：縮小 KV-cache 體積，同時守住吞吐\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>2. TQ4\u002F4 是最穩的預設值\u003C\u002Fh2>\u003Cp>作者明確把 TQ4\u002F4，也就是 4-bit K 加 4-bit V，當成預設生產設定。原因很直接：它在壓縮率、準確率和執行成本之間，給出最均衡的折衷。\u003C\u002Fp>\u003Cp>如果你要先選一個能落地的方案，這通常是起點，而不是終點。文章也指出 K 比 V 更敏感，所以 K 端用了 rotation 加 LUT quantization，V 端則維持較標準的 uniform quantization。\u003C\u002Fp>\u003Cul>\u003Cli>K 端：rotation + LUT quantization\u003C\u002Fli>\u003Cli>V 端：standard uniform quantization\u003C\u002Fli>\u003Cli>2-bit 方案可行，但額外成本較難回收\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>3. 軟注意力模型可先跳過邊界層\u003C\u002Fh2>\u003Cp>對 full-attention 模型來說，最簡單的精度補救之一，就是不要量化第一層和最後一層。這些 boundary layers 往往對 KV quantization 更敏感，保留全精度通常能換回一部分準確率。\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781299072582-zx9x.png\" alt=\"TurboQuant 在 AMD GPU 上把長上下文延遲壓下來\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>這個技巧不是通用規則。文章沿用 \u003Ca href=\"\u002Ftag\u002Fvllm\">vLLM\u003C\u002Fa> 的 heuristic，只把它用在 softmax attention 模型，不延伸到像 Qwen3.5 這類 hybrid attention 模型。對部署者來說，這是低風險、先試先贏的調整。\u003C\u002Fp>\u003Ccode>--kv-cache-dtype-skip-layers\n# softmax attention models 的邊界層可用\u003C\u002Fcode>\u003Ch2>4. Walsh-Hadamard 比隨機 rotation 更實用\u003C\u002Fh2>\u003Cp>原始 TurboQuant 允許 random rotation，但 ROCm 版本改偏向 Walsh-Hadamard transform，簡稱 WHT。原因不是理論包裝，而是它更適合 kernel，而且在實測中也有更好的表現。\u003C\u002Fp>\u003Cp>這個選擇同時改善了實作複雜度和效果。WHT 能把能量分散得更均勻，對 quantizer 更友善，也避開了 dense random rotation 在 production kernel 裡不好處理的問題。\u003C\u002Fp>\u003Cul>\u003Cli>比 random rotation 更好寫進 kernel\u003C\u002Fli>\u003Cli>在測試組合裡精度表現更佳\u003C\u002Fli>\u003Cli>方向上也和 TurboQuant+、llama.cpp 的做法一致\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>5. 4-bit 路徑先別加 QJL\u003C\u002Fh2>\u003Cp>作者對 QJL 的態度很乾脆：在 4-bit 預算下，它增加了複雜度和 runtime overhead，卻沒有帶來相稱的準確率收益。就他們的比較結果來看，直接不加 QJL 反而是最強的配置。\u003C\u002Fp>\u003Cp>他們也拆解了\u003Ca href=\"\u002Fnews\u002Fspacex-ipo-should-not-wash-away-grok-safety-failures-zh\">失敗\u003C\u002Fa>原因。原始 Gaussian projection matrix 表現最差；orthogonalized Gaussian 和 Walsh-Hadamard projection 可以補回不少，但放到 4-bit 的主路徑，最合理的做法仍是跳過 QJL。\u003C\u002Fp>\u003Cul>\u003Cli>raw Gaussian QJL 在 key 上最弱\u003C\u002Fli>\u003Cli>orthogonal-Gaussian 與 Walsh-Hadamard 可補回部分損失\u003C\u002Fli>\u003Cli>4-bit sweep 裡，單純 MSE 路徑勝過所有 K-side QJL 變體\u003C\u002Fli>\u003C\u002Ful>\u003Ch2>怎麼挑\u003C\u002Fh2>\u003Cp>如果你在做長上下文、多輪 \u003Ca href=\"\u002Ftag\u002Fagent\">agent\u003C\u002Fa> 服務，優先順序很清楚：先用 TQ4\u002F4，再加 WHT rotation，最後視模型型態加上 boundary-layer skipping。這組合最接近文章裡的生產最佳解。\u003C\u002Fp>\u003Cp>如果你的工作負載還沒到明顯記憶體瓶頸，或你對精度更敏感，就先留在 BF16 或 FP8，並把 TurboQuant 當成一套指引，專注處理最吃 KV-cache 的那一段。這篇真正的價值，是幫你判斷壓縮應該從哪裡開始。\u003C\u002Fp>","3.6x 加速、TTFT 13.9 秒降到 0.89 秒：這篇整理 TurboQuant 在 AMD GPU 上最值得採用的 5 個實作選擇。","rocm.blogs.amd.com","https:\u002F\u002Frocm.blogs.amd.com\u002Fartificial-intelligence\u002Fturboquant-vllm-agentic\u002FREADME.html",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781299071883-1324.png","industry","zh","093f7c46-be7c-4b62-be00-73808a61e0a0",[17,18,19,20,21,22,23,24],"TurboQuant","AMD GPU","ROCm","KV-cache","vLLM","long-context LLM","TQ4\u002F4","Walsh-Hadamard",[26,27,28],"ROCm 版 TurboQuant 把 KV-cache 壓縮做成可上線的 AMD serving 路徑，最高端到端加速達 3.6x。","TQ4\u002F4 是最穩的預設選擇，特別適合想先平衡壓縮率、準確率與延遲的部署。","WHT rotation、邊界層跳過和不加 QJL，是 4-bit 路徑裡最值得優先採用的三個決策。",2,"2026-06-12T21:17:25.608109+00:00","2026-06-12T21:17:25.602+00:00","caa87b65-9bbc-46fe-bba8-4f4158dd2d8b",{"tags":34,"relatedLang":46,"relatedPosts":50},[35,37,40,42,44],{"name":18,"slug":36},"amd-gpu",{"name":38,"slug":39},"KV cache","kv-cache",{"name":21,"slug":41},"vllm",{"name":19,"slug":43},"rocm",{"name":17,"slug":45},"turboquant",{"id":15,"slug":47,"title":48,"language":49},"turboquant-amd-gpus-kv-cache-latency-en","TurboQuant on AMD GPUs cuts KV-cache latency","en",[51,57,63,69,75,81],{"id":52,"slug":53,"title":54,"cover_image":55,"image_url":55,"created_at":56,"category":13},"78bb945b-f292-4071-811e-9ac390b68a38","anthropic-public-record-ai-anxiety-policy-zh","Anthropic 把 AI 焦慮變政策","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781327894646-6pyt.png","2026-06-13T05:17:42.429455+00:00",{"id":58,"slug":59,"title":60,"cover_image":61,"image_url":61,"created_at":62,"category":13},"a69174d1-9768-4144-909a-78ec2517b186","chatgpt-grew-from-chatbot-to-platform-zh","ChatGPT 從聊天機器人變平台","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781325173553-w7ov.png","2026-06-13T04:32:27.586497+00:00",{"id":64,"slug":65,"title":66,"cover_image":67,"image_url":67,"created_at":68,"category":13},"050bf93c-ddcf-4493-8335-11a67831fcfc","openai-files-confidential-ipo-after-122b-round-zh","OpenAI 密件申請 IPO，估值衝 8520 億","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781323369296-ra5z.png","2026-06-13T04:02:23.888945+00:00",{"id":70,"slug":71,"title":72,"cover_image":73,"image_url":73,"created_at":74,"category":13},"66a93d43-34f4-401b-b8a9-51878e91d60c","government-access-orders-frontier-model-access-zh","政府存取命令就該管住前沿模型存取","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781319763702-t9ak.png","2026-06-13T03:02:19.013704+00:00",{"id":76,"slug":77,"title":78,"cover_image":79,"image_url":79,"created_at":80,"category":13},"15b00407-d684-49c4-8b49-de247e4bbabe","6-kuan-ai-cheng-shi-dai-li-de-2026-fen-gong-zh","6 款 AI 程式代理的 2026 分工","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781317069290-tmbc.png","2026-06-13T02:17:21.790357+00:00",{"id":82,"slug":83,"title":84,"cover_image":85,"image_url":85,"created_at":86,"category":13},"5cb91c9d-9a8d-4e9f-a059-775982e25ddd","claude-design-partner-risk-zh","Claude Design 5 個教訓：合作先講會翻車","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1781316167850-1n3l.png","2026-06-13T02:02:20.840103+00:00",[88,93,98,103,108,113,118,123,128,133],{"id":89,"slug":90,"title":91,"created_at":92},"ee073da7-28b3-4752-a319-5a501459fb87","ai-in-2026-what-actually-matters-now-zh","2026 AI 真正重要的事","2026-03-26T07:09:12.008134+00:00",{"id":94,"slug":95,"title":96,"created_at":97},"83bd1795-8548-44c9-9a7e-de50a0923f71","trump-ai-framework-power-speech-state-preemption-zh","川普 AI 框架瞄準電力、言論與州權","2026-03-26T07:12:18.695466+00:00",{"id":99,"slug":100,"title":101,"created_at":102},"ea6be18b-c903-4e54-97b7-5f7447a612e0","nvidia-gtc-2026-big-ai-announcements-zh","NVIDIA GTC 2026 重點拆解","2026-03-26T07:14:26.62638+00:00",{"id":104,"slug":105,"title":106,"created_at":107},"4bcec76f-4c36-4daa-909f-54cd702f7c93","claude-users-spreading-out-and-getting-better-zh","Claude 用戶更分散，也更會用","2026-03-26T07:22:52.325888+00:00",{"id":109,"slug":110,"title":111,"created_at":112},"bd903b15-2473-4178-9789-b7557816e535","openclaw-raises-hard-question-for-ai-models-zh","OpenClaw 逼問 AI 模型價值","2026-03-26T07:24:54.707486+00:00",{"id":114,"slug":115,"title":116,"created_at":117},"eeac6b9e-ad9d-4831-8eec-8bba3f9bca6a","gap-google-gemini-checkout-fashion-search-zh","Gap 把結帳搬進 Gemini","2026-03-26T07:28:23.937768+00:00",{"id":119,"slug":120,"title":121,"created_at":122},"0740e53f-605d-4d57-8601-c10beb126f3c","google-pushes-gemini-transition-to-march-2026-zh","Google 把 Gemini 轉換延到 2026 年 3…","2026-03-26T07:30:12.825269+00:00",{"id":124,"slug":125,"title":126,"created_at":127},"e660d801-2421-4529-8fa9-86b82b066990","metas-llama-4-benchmark-scandal-gets-worse-zh","Meta Llama 4 分數風波又擴大","2026-03-26T07:34:21.156421+00:00",{"id":129,"slug":130,"title":131,"created_at":132},"183f9e7c-e143-40bb-a6d5-67ba84a3a8bc","accenture-mistral-ai-sovereign-enterprise-deal-zh","Accenture 攜手 Mistral AI 賣主權 AI","2026-03-26T07:38:14.818906+00:00",{"id":134,"slug":135,"title":136,"created_at":137},"191d9b1b-768a-478c-978c-dd7431a38149","mistral-ai-faces-its-hardest-year-yet-zh","Mistral AI 迎來最硬的一年","2026-03-26T07:40:23.716374+00:00"]