Zilliz adds cost-aware scoring to VDBBench

Zilliz updated VDBBench to score vector databases on cost as well as speed.

Zilliz has added cost-aware benchmarking to VectorDBBench, its open-source vector database benchmark. The update matters because teams choosing a vector database usually care about more than raw throughput; they also care about what the system will cost when it runs at scale.

The change is simple on paper and useful in practice: VDBBench now treats cost as a first-class metric instead of an afterthought. That gives buyers a way to compare databases on performance per dollar, which is often the number that survives the architecture review.

Why cost now belongs in vector DB tests

Vector databases sit in a strange spot in modern AI stacks. They are often chosen for retrieval-augmented generation, semantic search, recommendation systems, and other workloads where latency matters, but the bill matters too. A database that looks fast in a demo can become expensive once it is asked to handle real traffic, high availability, and large embedding sets.

That is why a cost-aware benchmark is more useful than a pure speed chart. It pushes teams to ask a better question: how much performance do we get for the money we spend? That question becomes especially important when a product team is deciding between managed services, self-hosted deployments, or a mix of both.

• Raw latency can hide infrastructure cost.
• Throughput numbers can look strong even when a system is pricey to run.
• Cost-aware scoring makes tradeoffs easier to compare across vendors.

What VDBBench is trying to fix

VDBBench is designed to be vendor-neutral, which is important because benchmark results lose value when they are tuned to favor one product. Zilliz says the project helps teams evaluate vector databases across real workload patterns instead of relying on marketing claims. That includes the kind of testing that matters for production planning, not just lab demos.

The new cost dimension also changes how the benchmark can be used inside procurement and engineering discussions. A platform team can compare systems with a clearer view of the operational bill, while application teams can see whether a cheaper option is actually good enough for their latency targets. In other words, the benchmark becomes a decision tool rather than a scoreboard.

"Benchmarks should measure what users actually pay for, not just what vendors want to show," said Zilliz founder and CEO Charles Xie in a statement tied to the VDBBench update.

That quote gets to the heart of the update. If a benchmark ignores cost, it can make expensive systems look better than they are for real deployments. Once cost enters the picture, the conversation shifts from bragging rights to operating reality.

How this changes vendor comparisons

Cost-aware benchmarking does not eliminate the need for careful testing, but it does sharpen the comparison. Two databases may return similar search quality and latency, yet one may need more memory, more replicas, or more expensive infrastructure to keep up. Those differences are easy to miss when the only visible number is queries per second.

For teams comparing vector database options, that means the analysis now has to include both technical and financial dimensions. The practical result is a more grounded shortlist, especially for companies that are scaling retrieval pipelines across multiple products.

• Milvus, Zilliz’s own vector database, is often evaluated in the same category as other vector engines.
• Pinecone is a managed option many teams compare against self-hosted systems.
• Weaviate is another common comparison point for semantic retrieval workloads.
• FAISS remains a widely used library for similarity search, especially in research and custom stacks.

The bigger implication is that benchmark culture is getting more realistic. AI infrastructure buyers have learned to distrust tidy performance charts that ignore cloud spend, storage overhead, and engineering time. A benchmark that puts cost on the same level as speed is a better fit for how these systems are actually purchased and operated.

What teams should do next

If your team is evaluating a vector database right now, this update is a reason to revisit your test plan. Run benchmarks that include latency, recall, throughput, and cost under the same workload. Compare the numbers at the scale you expect to run, not at a tiny proof-of-concept size.

The useful question is no longer which database is fastest in isolation. It is which one gives the best mix of speed, quality, and cost for your workload. That is the standard VDBBench is moving toward, and it is the standard more AI infrastructure teams should adopt before they sign a contract or spin up a cluster.

For readers tracking the open-source AI tooling stack, this update fits a broader pattern already visible in projects like real-workload LLM benchmarks: the industry is getting less interested in synthetic bragging rights and more interested in numbers that survive contact with production.