Tag
quantization
Quantization compresses model weights, activations, or KV cache into lower-bit formats to reduce memory and inference cost. Recent work spans 4-bit hybrid schemes and lower-bit LLM inference methods that target bottlenecks without sacrificing too much accuracy.
10 articles
UltraQuant: 4-bit KV caching for long agents
UltraQuant shows 4-bit KV caching can speed long, multi-turn agent serving while keeping more context resident.
llama.cpp’s latest release proves the project still wins by tightenin…
llama.cpp’s latest release shows that careful kernel fixes and backend tuning matter more than flashy features.
MLX Community lets you run Apple Silicon models
How MLX Community packages ready-to-run model weights for Apple Silicon with mlx-lm, mlx-vlm, and mlx-audio.
Unsloth’s Kimi-K2.5 GGUF pack lands on Hugging Face
Unsloth published GGUF quants of Kimi-K2.5 on Hugging Face, including 4-bit and 5-bit builds for local inference.
5 TurboQuant lessons for vector search teams
5 takeaways on Qdrant TurboQuant: how rotation changes compression, where recall holds up, and when safer quantizers fit better.
Shannon Scaling Law explains LLM overtraining
A Shannon-based scaling law explains why LLMs can get worse as compute rises under noise.
TurboQuant shows how 4-bit beats guesswork
I break down TurboQuant’s quantization study into a practical playbook for choosing 8-bit, 4-bit, PTQ, or QAT.
TurboQuant turns vLLM KV cache into 3-bit storage
I break down TurboQuant’s vLLM cache compression and give you a copy-ready setup for 3-bit KV cache and fallback paths.
TurboQuant cuts memory use 6x without accuracy loss
Google Research’s TurboQuant claims 6x less memory and 8x faster inference with no accuracy loss, jolting AI inference economics.
IF4: Smarter 4-Bit Quantization That Adapts to Your Data
MIT researchers propose a hybrid data format that switches between floating-point and integer representations, improving accuracy in 4-bit neural network quantization.