ExtractBench benchmarks schema-guided document extraction

Enterprise document extraction often fails in the same boring ways: fields get missed, records get truncated, or the output looks right but cannot be traced back to the source.

ExtractBench measures schema-guided enterprise document extraction across accuracy, grounding, and cost.

• Research org: Unspecified in arXiv abstract
• Core data: 4,869 pages
• Breakthrough: First benchmark to score accuracy, completeness, grounding, and cost together

That matters because “good extraction” is not just about getting a field value into JSON. In enterprise settings, teams also need the record to be complete, the output to follow the user’s schema, and the extracted data to carry source evidence that can survive review, debugging, and compliance checks.

What problem this paper is trying to fix

The paper focuses on schema-guided extraction: a document comes in, a user-defined schema goes out, and the system has to follow that schema faithfully while grounding its answers in the source. That is a more demanding task than plain OCR or generic information extraction, because the model is not just finding text — it is deciding what belongs in a structured record and proving where it came from.

For developers, the pain point is familiar. A system can look solid on a few easy documents and still fall apart when the document gets longer, the layout changes, or the schema requires many records instead of a single value. ExtractBench is built to make those failure modes visible instead of hiding them behind a single headline score.

The benchmark covers 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types. It also includes clear tags for different challenge scenarios, which is useful because not all extraction problems are equally hard. Short forms, long record lists, and other document types do not stress the same parts of a system.

How the benchmark is put together

ExtractBench combines three curation strategies to scale up ground truth without pretending every document can be handled the same way. For real documents, it uses independent-system agreement. For synthetic lists, it relies on known values. For forms, it adds human verification.

That mix is practical. It acknowledges that enterprise extraction data is expensive to label and that no single annotation method is enough for every document type. By combining methods, the authors can build a larger evaluation set while still keeping the labels credible enough for benchmarking.

The paper also says the benchmark is, to its knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. That is the key design choice here: instead of treating extraction as one metric problem, it treats it as a multi-objective system problem.

• Value accuracy is measured with order-insensitive value F1.
• Grounding is measured with word-level and page-level F1.
• Cost is measured alongside the quality metrics rather than ignored.

What the paper actually shows

The abstract does not give full benchmark tables or exact scores, so there are no detailed numbers to quote beyond the dataset scale and the ranking claims. What it does say is still enough to show the shape of the tradeoff.

Commercial VLMs perform well on short documents, but they often truncate record lists on long ones. That is an important practical warning: a model that seems strong on single-page or low-complexity inputs may not be reliable once the document demands full-list extraction.

Coding agents, by contrast, keep higher accuracy but at much higher cost. That is the classic enterprise tension: better output quality can come with a steep runtime or orchestration bill. ExtractBench makes that tradeoff visible instead of forcing teams to discover it in production.

The paper reports that LlamaExtract Agentic Plus ranks first on all three metrics, and that its accuracy is comparable to coding agents at a fraction of the cost. The abstract does not provide the exact metric values, so that is as far as the source lets us go. Still, the result suggests that agentic extraction systems may be able to close much of the quality gap without paying the full coding-agent price.

Why developers should care

If you are building document pipelines, vendor intake flows, compliance tooling, or any workflow that turns messy PDFs into structured records, this benchmark is useful because it tests the stuff that breaks real systems. A model that only scores well on value extraction is not enough if it drops records or cannot show its work.

Grounding is especially relevant for enterprise use. When an extracted field needs to be audited, reviewed, or reconciled, the system has to point back to the source text or page. ExtractBench’s word- and page-level grounding metrics give teams a way to compare systems on traceability, not just on final output shape.

The cost dimension matters too. Teams often compare models on quality alone and then discover that the “best” system is too expensive to run at scale. By putting cost into the evaluation, the benchmark pushes the conversation toward deployable systems rather than demo-friendly ones.

Limits and open questions

The abstract leaves several things unspecified. It does not provide the full metric tables, exact scores, or detailed per-category breakdowns in the summary we have here. It also does not tell us how the benchmark behaves across every challenge tag, so you should not assume the reported ranking generalizes uniformly across all document types.

Another limitation is that benchmark design can only approximate production reality. Even with 370 enterprise documents and multiple domains, real deployments may face different schemas, noisier scans, or more adversarial formatting. The benchmark is a strong step toward better evaluation, but it is still an evaluation set, not the whole problem.

Still, ExtractBench points in a direction that matters: extraction systems should be judged on whether they are accurate, complete, grounded, and affordable at the same time. That is a much better fit for enterprise workflows than a single accuracy number.

For teams choosing between commercial VLMs, coding agents, or more agentic extraction systems, the paper’s message is straightforward: short-document success is not enough, and cost cannot be an afterthought. A benchmark like this helps you ask the right question before you ship.

In that sense, ExtractBench is less about one model winning and more about changing what “good” means for document extraction. For developers, that is the useful part.