Kimi K3 Needs About 1.5 TB of Memory

Kimi K3 needs about 1.5 TB of memory, so most teams will find API pricing far cheaper than self-hosted GPU clusters.

Kimi K3 is a Moonshot AI model with 2.8 trillion total parameters, but its deployment math is shaped by much more than raw size. The article’s core claim is simple: if you want to run Kimi K3 yourself, you are planning around roughly 1.5 TB of memory, multiple high-end GPUs, and a very specific software stack.

That makes the real decision less about “can it run?” and more about “when does self-hosting beat API pricing?” The answer depends on token volume, concurrency, context length, and whether you need direct control over data and infrastructure.

Why Kimi K3 is hard to host locally

Kimi K3 uses a Mixture of Experts design, so each token activates only a slice of the model. The article says each token uses 16 of 896 experts plus two shared experts, which keeps inference cheaper than a dense model of similar size.

But sparse compute does not erase the storage problem. Kimi K3 ships with native MXFP4 weights, and the article estimates memory with a simple formula: total parameters multiplied by quantization bits, then divided by 8. On that basis, 2.8 trillion parameters at 4 bits comes out to about 1.4 TB for weights alone.

That number gets bigger once you add the rest of the inference stack. The article adds another 2 to 15 GB for KV cache, around 30 GB for activations, and about 30 GB for runtime overhead. The total lands near 1.5 TB for a single request path.

• Native FP4 weights reduce storage pressure compared with BF16 or FP16.
• Higher precision modes exist, but they are mainly for compatibility, not better output quality.
• Fine-tuning still needs higher precision, so FP4 is mostly an inference format.
• Kimi K2, by comparison, needs about 2 TB in BF16 for 1 trillion parameters.

The GPU count is only half the story

Once the memory math is clear, the hardware question becomes blunt: a single GPU cannot host Kimi K3. The article estimates that you would need about 19 NVIDIA H100 cards at 80 GB each, or about 11 NVIDIA H200 cards at 141 GB each, just to cover the base memory requirement.

That still does not mean those cards are the best fit. Hopper-class hardware and AMD MI300X can load 4-bit weights, but the article notes that they usually need to dequantize at runtime. That preserves some memory savings, yet it gives up part of the speed advantage built into Kimi K3’s native 4-bit design.

“The best way to run Kimi K3 is with 8 NVIDIA B300 or 8 AMD MI355X GPUs and tensor parallelism set to 8,” according to the vLLM deployment guidance referenced in the article.

That recommendation says a lot about where the model is headed in practice. The article argues that self-hosting Kimi K3 is really a hardware-generation decision, not just a capacity decision. If your stack cannot run native MXFP4 well, you are already paying an efficiency penalty.

There is also a software wrinkle. Kimi K3 uses a custom prefix-caching approach tied to its attention design, so standard vLLM prefix caching does not fit cleanly. Moonshot contributed a custom implementation, which means self-hosters need a newer vLLM build with K3-specific support.

Where the cost curve flips

The article’s pricing comparison is the part most teams will care about. Kimi K3’s official API pricing is $3 per million input tokens, $0.30 per million cached input tokens, and $15 per million output tokens. DigitalOcean Serverless Inference uses the same input and output prices, so the main difference is whether you pay per token or keep paying for GPUs all month.

For a native FP4 self-hosted setup, the article uses an 8-GPU GPU Droplet node built from AMD MI350X cards. At about $4.76 per GPU hour, the node costs roughly $38 per hour, or about $27,800 per month if it runs continuously.

That monthly bill is hard to justify unless your workload is very large and very steady. The article estimates that to match that spend through API output pricing alone, you would need about 1.8 billion output tokens per month, which works out to around 700 tokens per second every hour of the month.

• One heavy user at 50 million tokens per month would pay about $750 through the API if all tokens were billed as output.
• The same user would be far below the utilization needed to justify a dedicated 8-GPU node.
• The article says API use is about 40 times cheaper for a single heavy user under its assumptions.
• Self-hosting starts to make more sense once you have more than 40 steady heavy users.

That last point matters because “40 users” is really shorthand for sustained throughput. It could be 40 people, or it could be one engineer running a swarm of agents that keep the node busy all day.

Concurrency changes the math, but not enough for one user

The article also explains why a single user cannot simply rent fewer GPUs and call it a day. Tensor parallelism works best on 2, 4, or 8 GPUs, not odd counts like 6, and most providers sell these machines in fixed blocks anyway.

Even if you could assemble a smaller set, token output rates would still be the bottleneck. A single request rarely produces enough tokens per second to keep a 6-GPU setup busy enough to offset its rental cost.

What does help is multi-request concurrency. The article says one node can handle about 40 parallel requests, and with shorter contexts the number of resident requests can rise sharply because KV cache usage drops.

• Full 1 million token context: about 15 GB per request, around 40 requests per 600 GB pool.
• 128K context: about 2 GB per request, around 300 requests.
• 32K context: about 1.5 GB per request, around 400 requests.
• 8K context: about 1 GB per request, around 600 requests.

That is the real tradeoff. Kimi K3 becomes more attractive when you can keep the node busy with many requests, many agents, or long-lived enterprise workloads. If you only have one or two users, the API is still the cleaner financial choice.

What this means for teams choosing a deployment model

There is a practical split here. If you want the simplest path, token-based API access wins. You avoid buying expensive GPUs, you skip model serving work, and you do not need to maintain a custom inference stack.

If you need tighter control over data residency, latency consistency, or model ownership, self-hosting starts to look better. The article also points out that some third-party providers do not store inference data, so privacy concerns alone do not automatically force a self-hosted setup.

Licensing matters too. Kimi K3 does not use Apache or MIT licensing. It uses a custom license that allows use, modification, fine-tuning, deployment, distribution, and commercialization, but it adds conditions for large-scale model-as-a-service businesses and very large commercial products.

For teams evaluating Kimi K3 deployment costs, the takeaway is straightforward: compare token volume, not just token price. If your workload is bursty, the API is probably cheaper. If your workload is steady, multi-user, and GPU-efficient, dedicated infrastructure can eventually win.

The more interesting question is what happens when more open-weight models arrive with similar FP4 support. If vendors keep shipping native low-precision inference paths, the gap between “cheap to call” and “cheap to own” will keep narrowing, but the break-even point will still be determined by utilization, not hype.