[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-ultraquant-4bit-kv-caching-agents-en":3,"article-related-ultraquant-4bit-kv-caching-agents-en":30,"series-research-9fd702bc-6c80-4d27-8f85-5971f898bef3":78},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":29},"9fd702bc-6c80-4d27-8f85-5971f898bef3","ultraquant-4bit-kv-caching-agents-en","UltraQuant: 4-bit KV caching for long agents","\u003Cp data-speakable=\"summary\">UltraQuant shows 4-bit KV caching can speed long, multi-turn \u003Ca href=\"\u002Ftag\u002Fagent\">agent\u003C\u002Fa> serving while keeping more context resident.\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>Research org\u003C\u002Fstrong>: Advanced Micro Devices + UCLA + Purdue University\u003C\u002Fli>\u003Cli>\u003Cstrong>Core data\u003C\u002Fstrong>: 3.47× P50 TTFT improvement in late rounds\u003C\u002Fli>\u003Cli>\u003Cstrong>Breakthrough\u003C\u002Fstrong>: FP4 KV tensors with UE8M0 scales on CDNA4 scaled-MFMA\u003C\u002Fli>\u003C\u002Ful>\u003Cp>Long-context agents are a memory problem as much as a model problem. When a session stretches across many turns, the \u003Ca href=\"\u002Ftag\u002Fkv-cache\">KV cache\u003C\u002Fa> grows, fills HBM, and starts pushing useful context out of device memory. This paper looks at what happens when you compress that cache to 4 bits without breaking serving performance.\u003C\u002Fp>\u003Cp>The practical question is not just “can we shrink KV state?” It is “can we shrink it in a way that still works for agentic workloads, where the same prefix is reused over and over and concurrency matters?” UltraQuant argues that the answer is yes, but only if you treat quality, cache residency, and kernel efficiency as one system.\u003C\u002Fp>\u003Ch2>What problem this paper is trying to fix\u003C\u002Fh2>\u003Cp>Modern LLM agents do more than chat. They browse, inspect repositories, invoke tools, and carry a long working memory across many rounds. That means the KV cache can become a first-order consumer of high-bandwidth memory, especially as context windows move toward one million tokens and beyond.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782331384598-tjhi.png\" alt=\"UltraQuant: 4-bit KV caching for long agents\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The paper frames this as a serving bottleneck. If the cache is too large, the system evicts useful prefixes and has to re-prefill them later. If compression introduces too much overhead, the model may save memory but lose latency. So the real target is not raw compression alone; it is preserving resident context while keeping TTFT and TPOT under control.\u003C\u002Fp>\u003Cp>UltraQuant is positioned against two anchors in the paper: \u003Ca href=\"\u002Ftag\u002Fturboquant\">TurboQuant\u003C\u002Fa>-style 4-bit quantization as a quality anchor, and vLLM FP8 KV caching as a deployment anchor. FP8 already gives roughly 2× compression with near-lossless quality and native hardware support, so a 4-bit scheme has to justify itself beyond just smaller tensors.\u003C\u002Fp>\u003Ch2>How the method works in plain English\u003C\u002Fh2>\u003Cp>The paper studies two related paths. The first is Ultra-TurboQuant, or Ultra-TQ, which keeps the TurboQuant representation but improves the implementation. The second is UltraQuant, which goes a step further and replaces the codebook with a hardware-native FP4 approximation path.\u003C\u002Fp>\u003Cp>Ultra-TQ starts from the same basic idea as TurboQuant: rotate the KV vectors so outliers are spread across channels, making the distribution easier to quantize. The paper uses a Walsh–Hadamard rotation and removes QJL, while also making keys and values asymmetric because the two behave differently under quantization.\u003C\u002Fp>\u003Cp>One practical tweak the paper emphasizes is calibrated centroids. Instead of relying only on theoretical Lloyd–Max centroids, it refits the 16-entry table on captured activations. The paper says this is cheap: a single forward pass over about 20 sampled vectors per rotated layer, and in the real implementation it is only applied to the 10% of layers with higher per-element quantization MSE.\u003C\u002Fp>\u003Cp>UltraQuant then swaps the codebook path for an FP4 micro-tensor approach. The abstract says this uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on CDNA4. In other words, the design tries to make dequantization disappear into the matrix core instead of paying extra software lookup costs.\u003C\u002Fp>\u003Cp>That matters because codebook quantization can be accurate but still awkward for serving. The paper explicitly calls out lookup and irregular-access overhead as a weakness of codebook-based methods. UltraQuant’s bet is that a hardware-native format is easier to run at speed than a more exact but less convenient representation.\u003C\u002Fp>\u003Ch2>What the paper actually shows\u003C\u002Fh2>\u003Cp>The evaluation is built around an agentic workload rather than a single-turn \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa>. The paper uses vLLM’s native multi-turn benchmark with ShareGPT conversation data, serves 32 concurrent chat sessions, and reports P50 TTFT and P50 TPOT. That setup is meant to simulate cache pressure in long-running sessions where prefixes get reused across turns.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782331379574-4la6.png\" alt=\"UltraQuant: 4-bit KV caching for long agents\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>On a long-context, multi-turn agentic workload, UltraQuant reports a 3.47× improvement in P50 TTFT in the late rounds, where the cache is most pressured. Across all rounds, the TTFT improvement is 2.3×, and output throughput rises by 1.63× over the FP8 KV baseline.\u003C\u002Fp>\u003Cp>The paper also notes an important nuance: UltraQuant is not uniformly faster in every phase. In the warm rounds, the reported P50 TTFT is 0.86× relative to FP8 KV, which means FP8 is faster there. The advantage shows up later, when long per-client prefixes exceed the effective resident-cache capacity of FP8 and UltraQuant keeps more prefixes on device.\u003C\u002Fp>\u003Cp>That makes the result more interesting than a simple compression story. The improvement is attributed to cache residency rather than re-prefill, and the per-round breakdown shows UltraQuant keeping both TTFT and TPOT low across all six conversation rounds while FP8 degrades as context accumulates.\u003C\u002Fp>\u003Cp>The paper also reports an accuracy-related result for calibrated centroids. Compared with an apples-to-apples fakequant control, refitting the codebook lowers per-element K quantization MSE by 10.3%, from 1.32×10^-4 to 1.18×10^-4. The abstract does not give a full task-accuracy table in the excerpt provided, so this MSE result is the clearest concrete quality number available here.\u003C\u002Fp>\u003Ch2>What developers should take away\u003C\u002Fh2>\u003Cp>If you are building or serving context-heavy agents, this paper is a reminder that KV-cache design is part of the product, not just a backend detail. A 4-bit scheme only helps if it survives the full path from quantization to kernel execution to multi-turn serving behavior.\u003C\u002Fp>\u003Cp>The strongest engineering lesson is that compression format and deployment format are not the same thing. TurboQuant-style codebooks may be a good algorithmic anchor, but if they force expensive lookup or dequantization, the serving system may give back the gains. UltraQuant’s approach is to align the representation with the hardware path instead of treating them separately.\u003C\u002Fp>\u003Cp>Another useful takeaway is that the best metric is workload-specific. This paper does not present a generic “one benchmark to rule them all” claim. It focuses on long-context, concurrent, multi-turn agent sessions, which is exactly where cache residency and latency interact in messy ways. If your workload is mostly short prompts, the tradeoff may look different.\u003C\u002Fp>\u003Ch2>Limitations and open questions\u003C\u002Fh2>\u003Cp>The abstract and notes provided do not include a broad set of benchmark numbers beyond the agentic serving results and the quantization MSE example. That means readers should be careful about extrapolating the gains to other models, other workloads, or other hardware.\u003C\u002Fp>\u003Cp>The paper is also explicit that its gains are tied to AMD Instinct GPUs and CDNA4 native support. UltraQuant’s FP4 path depends on hardware features such as scaled-MFMA, so the implementation story may not transfer cleanly to other accelerators.\u003C\u002Fp>\u003Cp>There is also a tradeoff visible in the results themselves: the warm rounds do not show the same benefit as the cache-pressured late rounds. That suggests the win comes from reducing eviction and keeping more context resident, not from a universal per-\u003Ca href=\"\u002Ftag\u002Ftoken\">token\u003C\u002Fa> speedup.\u003C\u002Fp>\u003Cp>For developers, the main question is whether your serving stack is already bottlenecked by resident KV capacity. If it is, UltraQuant points to a path where 4-bit caching is not just a memory optimization but a latency optimization for real agent workloads. If not, the payoff may be smaller than the headline numbers suggest.\u003C\u002Fp>\u003Ch2>Bottom line\u003C\u002Fh2>\u003Cp>UltraQuant is a serving-first take on 4-bit KV caching for long-running agents. It combines quantization choices, cache layout, and AMD GPU kernel support to make compressed KV state practical in a multi-turn setting.\u003C\u002Fp>\u003Cp>The paper’s core message is simple: for context-heavy agents, the right 4-bit cache can improve both memory residency and end-to-end serving speed, but only when the representation and the hardware path are designed together.\u003C\u002Fp>\u003Cul>\u003Cli>4-bit KV caching is only useful if it helps resident context, not just tensor size.\u003C\u002Fli>\u003Cli>UltraQuant’s strongest gains show up in late, cache-pressured rounds.\u003C\u002Fli>\u003Cli>The paper’s main contribution is a hardware-aware path from quantization to serving.\u003C\u002Fli>\u003C\u002Ful>","UltraQuant shows 4-bit KV caching can speed long, multi-turn agent serving while keeping more context resident.","arxiv.org","https:\u002F\u002Farxiv.org\u002Fhtml\u002F2606.20474v1",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782331384598-tjhi.png","research","en","a2242009-98d7-409c-9f22-d825a81fef2e",[17,18,19,20,21],"KV cache","quantization","agent serving","FP4","vLLM",[23,24,25],"UltraQuant targets long-context agent serving, not just smaller KV tensors.","Its biggest gains appear when cache pressure is high in later conversation rounds.","The paper ties 4-bit quantization to AMD GPU-native execution, not just algorithm design.",0,"2026-06-24T20:02:33.028079+00:00","2026-06-24T20:02:33.02+00:00","3103988e-c4fe-45e3-98ab-846500c9d507",{"tags":31,"relatedLang":37,"relatedPosts":41},[32,34,35],{"name":17,"slug":33},"kv-cache",{"name":18,"slug":18},{"name":21,"slug":36},"vllm",{"id":15,"slug":38,"title":39,"language":40},"ultraquant-4bit-kv-caching-agents-zh","UltraQuant：4-bit KV 快取加速長代理","zh",[42,48,54,60,66,72],{"id":43,"slug":44,"title":45,"cover_image":46,"image_url":46,"created_at":47,"category":13},"06b86f04-a846-4cd5-95f0-1a5d3925c846","opsd-user-feedback-training-loop-en","OPSD lets you turn user clicks into training","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782335103738-zb9h.png","2026-06-24T21:04:40.861287+00:00",{"id":49,"slug":50,"title":51,"cover_image":52,"image_url":52,"created_at":53,"category":13},"67326f4b-c9f1-4c67-ad20-69bf93134fc1","flux3d-3d-gaussian-generation-diffusion-en","FLUX3D fixes 3DGS detail loss from images","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782284584653-nrlg.png","2026-06-24T07:02:37.868681+00:00",{"id":55,"slug":56,"title":57,"cover_image":58,"image_url":58,"created_at":59,"category":13},"59a57ebc-6f6e-4454-9cd2-51fca86a6a26","stochastic-subgradient-last-iterate-bounds-en","Stochastic Subgradient Last Iterate Gets Tight Bounds","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782283673230-gyie.png","2026-06-24T06:47:29.673643+00:00",{"id":61,"slug":62,"title":63,"cover_image":64,"image_url":64,"created_at":65,"category":13},"d3e6b375-22a5-476f-87bb-df3751552e24","insight-vla-self-guided-skill-acquisition-en","InSight lets VLAs learn new skills on their own","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782282778691-9enz.png","2026-06-24T06:32:31.387158+00:00",{"id":67,"slug":68,"title":69,"cover_image":70,"image_url":70,"created_at":71,"category":13},"fed4d40e-4605-4ce8-b5be-fccfded84eea","anthropic-right-alarm-recursive-self-improvement-en","Anthropic is right to sound the alarm on recursive self-improvement","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782263866756-axdv.png","2026-06-24T01:17:21.01479+00:00",{"id":73,"slug":74,"title":75,"cover_image":76,"image_url":76,"created_at":77,"category":13},"3d56760e-635e-4e72-905d-c3afff8cda2e","openai-bug-hunt-chrome-safari-firefox-en","OpenAI’s bug hunt rattled Chrome, Safari, Firefox","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782258470980-462a.png","2026-06-23T23:47:31.141534+00:00",[79,84,89,94,99,104,109,114,119,124],{"id":80,"slug":81,"title":82,"created_at":83},"a2715e72-1fe8-41b3-abb1-d0cf1f710189","ai-predictions-2026-big-changes-en","AI Predictions for 2026: Brace for Big Changes","2026-03-26T01:25:07.788356+00:00",{"id":85,"slug":86,"title":87,"created_at":88},"8404bd7b-4c2f-4109-9ec4-baf29d88af2b","ml-papers-of-the-week-github-research-desk-en","ML Papers of the Week Turns GitHub Into a Research Desk","2026-03-27T01:11:39.480259+00:00",{"id":90,"slug":91,"title":92,"created_at":93},"87897a94-8065-4464-a016-1f23e89e17cc","ai-ml-conferences-to-watch-in-2026-en","AI\u002FML Conferences to Watch in 2026","2026-03-27T01:51:54.184108+00:00",{"id":95,"slug":96,"title":97,"created_at":98},"6f1987cf-25f3-47a4-b3e6-db0997695be8","openclaw-agents-manipulated-self-sabotage-en","OpenClaw Agents Can Be Manipulated Into Failure","2026-03-28T03:03:18.899465+00:00",{"id":100,"slug":101,"title":102,"created_at":103},"a53571ad-735a-4178-9f93-cb09b699d99c","vega-driving-language-instructions-en","Vega: Driving with Natural Language Instructions","2026-03-28T14:54:04.698882+00:00",{"id":105,"slug":106,"title":107,"created_at":108},"a34581d6-f36e-46da-88bb-582fb3e7425c","personalizing-autonomous-driving-styles-en","Drive My Way: Personalizing Autonomous Driving Styles","2026-03-28T14:54:26.148181+00:00",{"id":110,"slug":111,"title":112,"created_at":113},"2bc1ad7f-26ce-4f02-9885-803b35fd229d","training-knowledge-bases-writeback-rag-en","Training Knowledge Bases with WriteBack-RAG","2026-03-28T14:54:45.643433+00:00",{"id":115,"slug":116,"title":117,"created_at":118},"71adc507-3c54-4605-bbe2-c966acd6187e","packforcing-long-video-generation-en","PackForcing: Efficient Long-Video Generation Method","2026-03-28T14:55:02.646943+00:00",{"id":120,"slug":121,"title":122,"created_at":123},"675942ef-b9ec-4c5f-a997-381250b6eacb","pixelsmile-facial-expression-editing-en","PixelSmile Framework Enhances Facial Expression Editing","2026-03-28T14:55:20.633463+00:00",{"id":125,"slug":126,"title":127,"created_at":128},"6954fa2b-8b66-4839-884b-e46f89fa1bc3","adaptive-block-scaled-data-types-en","IF4: Smarter 4-Bit Quantization That Adapts to Your Data","2026-03-31T06:00:36.65963+00:00"]