vLLM, SGLang, vMLX: better local LLM runtimes

vLLM, SGLang, vMLX, MLC-LLM, and ExLlamaV3 target serious local LLM workflows beyond Ollama and llama.cpp.

Most people start local LLMs with Ollama or llama.cpp, and that still makes sense. But as soon as a model becomes part of a real workflow, the runtime matters as much as the model, especially for serving, batching, cache behavior, and hardware-specific acceleration.

What changed

The article argues that the local AI stack has split into specialized tools for different jobs. Instead of one default runtime, developers now choose based on whether they need an inference server, a Mac-native app layer, browser or mobile deployment, or better performance on consumer GPUs.

Here are the main alternatives highlighted:

• vLLM for high-throughput serving, OpenAI-compatible APIs, continuous batching, and PagedAttention.
• SGLang for structured generation, repeated prompt patterns, tool use, and cache reuse.
• MLX and MLX-LM, plus vMLX, for Apple Silicon workflows.
• MLC-LLM and WebLLM for browsers, phones, tablets, and embedded targets.
• ExLlamaV3 for consumer GPU inference, with TabbyAPI for OpenAI-style serving.

vLLM is positioned as the first step up when a local model needs to act like infrastructure. Its batching and cache management are aimed at multiple apps or agents hitting the same endpoint, not just one person chatting in a terminal.

SGLang goes after similar workloads but with more emphasis on structured output. The article notes support for RadixAttention, prefill-decode disaggregation, speculative decoding, tensor and expert parallelism, and multi-LoRA batching, all aimed at repeated prompts and schema-driven responses.

Why it matters

For developers, the shift is practical: once a model backs tools, agents, RAG experiments, or multiple clients, the choice of runtime can change latency, VRAM use, and output reliability. A local LLM that only answers prompts is easy to run; a local LLM that must serve APIs and return valid JSON is a different problem.

The market effect is also clear. Local AI is no longer one-size-fits-all, and the stack is fragmenting around hardware and deployment target. Mac users get native paths, Nvidia users get optimized serving, AMD gets its own tooling, and consumer GPUs get runtimes tuned to fit memory limits instead of enterprise assumptions.

The takeaway is simple: Ollama and llama.cpp are still the easy defaults, but serious local AI work now starts with a question about the runtime, not just the model.