Why tool calling may work better as code

A BFCL v4 study finds programmatic tool calling often beats JSON tool calls across 14 models.

• Research org: Unspecified in arXiv abstract
• Core data: 10.6% improvement over JSON baseline
• Breakthrough: Expose tools as typed Python stubs executed in one agent turn

The Bitter Lesson of Tool Calling looks at a simple but important question for agent builders: should an LLM call tools through rigid JSON, or should it write code to do it? The paper argues that for code-capable models, treating tools as code can be a better fit for real agent workflows, especially when tasks need chaining or parallel execution.

That matters because tool calling is one of the core mechanics behind modern agents. If the interface is brittle, the agent can fail even when the model itself is capable. If the interface matches how the model already reasons about code, you may get better behavior without changing the underlying model.

What problem this paper is trying to fix

Most production tool use still relies on structured JSON calls. That approach is easy to validate and integrate, but it can be awkward when an agent needs to chain multiple tool calls, fan out work in parallel, or manage execution flow more naturally.

The paper frames this as a gap between what code-capable models can do and what the tool interface allows them to express. Instead of forcing the model into a rigid call format, the authors test whether exposing tools as typed Python stubs gives the model a more natural way to act.

Importantly, the paper says there has not been a systematic evaluation of tools-as-code against native JSON calling across current and prior model generations under real-world task conditions. This study is meant to fill that gap, at least on one benchmark: BFCL v4.

How the method works in plain English

In programmatic tool calling, the model does not emit a JSON blob and wait for a separate tool router to interpret it. Instead, tools are exposed as typed Python stubs, and the model invokes them through code. Execution and results are handled in a single agent turn.

That design changes the shape of the interaction. A model can chain calls in code, parallelize work more naturally, and express control flow in a way that looks closer to ordinary programming. The paper does not claim this is universally superior; it tests whether the interface itself helps on BFCL v4.

This is a useful framing for engineers because it moves the discussion away from “which prompt is better” and toward “which execution model matches the task.” For code-heavy agents, the interface can be as important as the model.

What the paper actually shows

The authors compare programmatic tool calling, or PTC, against native JSON tool calling across 14 language models on BFCL v4. The headline result is that PTC matches or exceeds JSON calling in 11 of those 14 models.

The strongest result mentioned in the abstract is for the GPT-5.6 family, which shows a 10.6% improvement over the JSON tool-calling baseline. The abstract does not provide the full per-model table, so you should treat that number as the clearest concrete result rather than a complete picture of every model tested.

The paper also tests a parallel fan-out setting, where PTC matches or outperforms the baseline in 13 of 14 models. That is a meaningful signal for agent workloads that need to split work across multiple tools or sub-tasks instead of handling everything sequentially.

Another interesting result is robustness under context rot conditions. The abstract says the baseline degrades by 2.3% on average, while PTC holds stable. The paper does not give more detail in the abstract about the exact setup behind context rot, so the safe takeaway is that PTC appears less sensitive to that failure mode in the tested conditions.

• PTC matched or beat JSON calling in 11 of 14 models on BFCL v4.
• GPT-5.6 family improved by 10.6% over the JSON baseline.
• Under parallel fan-out, PTC matched or beat baseline in 13 of 14 models.
• Under context rot, the baseline degraded by 2.3% on average.

Why developers should care

If you are building an agent stack, this paper suggests the tool interface is not a minor implementation detail. For code-capable models, letting the model call typed Python stubs may be a better default than forcing every action through JSON.

That could simplify agent design in a few practical ways. Code-based tool use may fit workflows that require branching, loops, fan-out, or staged execution. It may also reduce the mismatch between the model’s internal “thinking in code” and the external contract it has to follow.

Still, the paper is careful not to overclaim. The results are reported on BFCL v4, not across every possible agent benchmark or production environment. The abstract also does not include latency, cost, reliability under malformed inputs, or implementation overhead, so those questions remain open.

For practitioners, the most useful reading is not “JSON is dead.” It is that programmatic tool calling looks like a credible alternative worth testing when your models are already code-capable and your tasks benefit from richer control flow.

What is still missing

The abstract gives a clear directional result, but it leaves out a lot of operational detail. There is no benchmark table in the source text beyond the headline percentages, no discussion of failure cases, and no breakdown of how much of the gain comes from the interface versus model-specific behavior.

There is also an important scope note: the paper says PTC tracks model capability across release generations. That suggests the interface’s benefits may depend on the model family and its coding competence, rather than being a universal replacement for JSON tool calling.

So the practical takeaway is measured, not dramatic. If you are shipping agents, this paper is a strong reason to evaluate tools-as-code in your own stack, especially for code-capable models and workflows with parallel or multi-step tool use.