OpenSearch’s vector search benchmark in 5 parts

How do you benchmark OpenSearch vector search without redoing the same index build every time?

OpenSearch vector search benchmarks indexing, training, search, and merge behavior in five test procedures.

1. No-train test

The no-train test is the default fit for vector methods that do not require a training phase. It lets you define the engine, space type, and other method settings, then measure how the index behaves under load.

Use this when you want to compare indexing and query behavior across configurations without adding a training step to the workflow. The workload supports datasets in HDF5 or BIG-ANN format, which makes it easier to reuse the same corpus across runs.

• Best for methods that index directly
• Useful for comparing engine and space-type choices
• Works with both HDF5 and BIG-ANN datasets

2. No-train test, index only

This variant isolates ingestion. If your question is “How fast can this vector index be built?”, this is the procedure to use. It removes search from the picture so indexing throughput and merge behavior are easier to read.

That makes it a good fit for bulk-load tuning, shard sizing, and client-count experiments. The workload parameters include target_index_bulk_size, target_index_bulk_index_clients, and target_index_dimension, so you can shape the ingest path with some precision.

3. No-train test for Amazon OpenSearch Serverless

The Serverless variant is aimed at vector search collections in Amazon OpenSearch Serverless. It uses the same no-train idea, but it follows the limits of Serverless collections instead of the full cluster workflow.

One important difference is that it does not include refresh and warmup operations, because those are not supported for vector search collections. If you are validating a Serverless deployment, this keeps the benchmark aligned with what the platform can actually do.

• Use it only for Serverless vector search collections
• No refresh step
• No warmup step

4. Force merge index procedure

The force merge procedure measures the cost of compacting vector indexes down to a target segment count. For large datasets, that cost can be high, so the docs recommend running it as a separate procedure instead of folding it into every benchmark.

This is the right choice when you need to know how much segment reduction affects later search behavior or storage layout. The workload exposes target_index_max_num_segments and target_index_force_merge_timeout, which makes the merge target and wait time explicit.

• Targets segment count before search
• Useful for occasional maintenance benchmarks
• Can be expensive on large corpora

5. Search procedure

The search procedure benchmarks already indexed vector search data. That matters when reindexing would waste time, especially for large datasets where load time can be substantial.

It also includes warmup operations to reduce cold-start effects, so the result is closer to steady-state query performance. The sample output in the docs shows a prod-queries median latency of 3.43393 ms and a mean throughput of 213.85 ops/s, which gives a concrete sense of the numbers this workload can produce.

How to decide

Pick no-train if your vector method does not need training and you want a general benchmark. Pick index only if ingestion speed is the main question, and choose the Serverless variant if you are testing OpenSearch Serverless collections.

If you care about segment compaction, use force merge. If the index already exists and you want query latency or throughput, use search so you can avoid rebuilding the data each run.