onepot-Bench 0 tests lab-aware chemistry models

onepot-Bench 0 tests chemistry models on lab-relevant reasoning, safety, and private wet-lab prediction tasks.

• Research org: Unspecified in arXiv abstract
• Core data: No benchmark numbers in abstract
• Breakthrough: Three-part benchmark: ChemAbacus, SynthRefusal, and SynthBench

Most model evaluations still reward generic problem solving, public-data recall, or narrow task accuracy. This paper argues that is not enough for chemistry, where a model also has to handle lab constraints, safety boundaries, and decisions that actually matter at the bench.

To address that gap, the authors introduce onepot-Bench 0: towards lab-aware in silico chemistry benchmarks, a proprietary benchmark suite aimed at synthetic chemistry capabilities relevant to wet-lab execution. The point is not just to see whether a model can answer chemistry questions, but whether it can make decisions that are more aligned with real laboratory work.

What problem this paper is trying to fix

The paper starts from a practical complaint: current evaluations do a poor job of measuring the abilities needed for reliable decisions in a physical laboratory. In other words, a model might look good on a public benchmark and still be brittle when asked to help with experiment planning, execution, or post-hoc analysis.

That mismatch matters because scientific work is not a single skill. The abstract says these capabilities require a mixture of problem-solving and domain-specific intuition. A benchmark that only checks one of those pieces can miss the failure modes that matter in real lab settings.

There is also a data-quality issue. The authors say existing evaluations often rely on public data that may have appeared in model training corpora. If a model has already seen the answers, the benchmark can overstate what it actually knows. onepot-Bench 0 is positioned as a way to reduce that problem by using private experimental data generated in the authors’ laboratory for part of the suite.

How the benchmark is organized

onepot-Bench 0 is made of three complementary evaluations. Each one looks at a different layer of chemistry competence, from basic literacy to deeper reaction-specific knowledge.

ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning. That phrasing matters: the benchmark is not asking the model to call external tools or lean on a calculator pipeline. It is checking whether the model can work through chemistry-flavored numbers and representations on its own.

SynthRefusal characterizes safety and refusal behavior across benign, controlled, and designer-drug targets. That means the benchmark is not just about getting the “right” answer; it is also about whether the model knows when to refuse or stay within appropriate bounds.

SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in the lab. This is the most directly wet-lab-oriented piece of the suite, because it asks the model to predict outcomes and choose catalysts in a setting tied to real experimental work.

Together, these three parts probe what the authors call basic competency, reliability, and deeper knowledge. That framing is useful for engineers because it separates “can the model reason at all?” from “can the model be trusted around lab decisions?”

What the paper actually shows

From the abstract alone, the paper does not report benchmark scores, accuracy numbers, or comparative wins over prior systems. So there are no concrete performance metrics to summarize here.

What the abstract does show is the benchmark design itself. The authors are making a methodological argument: if you want to evaluate chemistry models for lab use, you need tests that cover tool-free reasoning, refusal behavior, and private-data prediction tasks, not just generic chemistry QA.

That is a meaningful contribution even without numbers in the abstract. Benchmarks often shape what model builders optimize for, and in this case the benchmark is explicitly trying to steer evaluation toward laboratory realism rather than public-data familiarity.

Why developers and researchers should care

If you are building or integrating language models into chemistry workflows, the main lesson is that “good on benchmarks” is not the same thing as “safe and useful in the lab.” onepot-Bench 0 is a reminder that deployment-oriented evaluation needs to include refusal behavior, numerical reasoning, and tasks grounded in experimental reality.

For developers, that has a few practical implications. First, you may need separate checks for basic chemistry literacy and for reaction-level prediction. Second, safety should be evaluated as a first-class behavior, not an afterthought. Third, if your evaluation set is public, you should assume contamination is possible and treat the result cautiously.

The use of private experimental data is especially important. It suggests a path toward benchmarks that are harder to memorize and more representative of real lab work. But it also means reproducibility and external comparison may be more limited than with fully public datasets, at least based on what the abstract tells us.

Limitations and open questions

The biggest limitation here is simple: the abstract does not provide results. We do not know how hard the benchmark is, how models performed, or whether any particular model struggled more with refusal, reasoning, or reaction prediction.

Another open question is how broadly the benchmark generalizes. Because one part uses private experimental data generated in the authors’ laboratory, outsiders will want to know how representative that data is of other chemistry settings and whether the benchmark can be compared across labs.

There is also a scope question. The suite is aimed at synthetic chemistry capabilities relevant to wet-lab execution, which is a focused target. That focus is a strength, but it also means the benchmark is not a full measure of all scientific reasoning. It is a lab-aware chemistry benchmark, not a universal science benchmark.

Even so, the paper points in a useful direction. Instead of asking whether a model can merely talk about chemistry, it asks whether it can support decisions that survive contact with the lab. For anyone building model-assisted scientific tools, that is the evaluation gap that matters most.

• onepot-Bench 0 splits chemistry evaluation into literacy, refusal, and lab-grounded prediction.
• The suite tries to reduce public-data leakage by using private experimental data for SynthBench.
• The abstract does not include scores, so the main contribution is the benchmark design itself.