The Bitter Lesson of Tool Calling | AI Paper Digest

TL;DR Highlight

Calling tools via Python scripts instead of JSON matched or beat JSON on 11 of 14 models.

Who Should Read

Backend developers building agents that chain or fan out many tool calls and hit turn-count or dropped-call problems

Core Mechanics

Code-style calls tie or win on 11/14 models
- Programmatic tool calling (PTC) exposes tools as typed Python stubs and lets the model write a script to call them.
- On a 309-entry BFCL v4 subset, 11 models matched or beat JSON tool calling.
Model generation mattered, not vendor
- All five Claude models and the newest GPT-5.6 models matched or beat JSON.
- GPT-4o, GPT-4.1 and GPT-5.4-mini emitted literal \n instead of newlines and hit syntax errors.
Longer chains favor code
- At chain length 12 or more, PTC was 18.8% absolute higher than JSON.
- JSON pays an extra inference turn per link.
JSON dropped calls at high fan-out
- Claude Sonnet 5 began dropping calls around N=70-72 with JSON.
- PTC kept 100% enumeration accuracy at N=100.
PTC held up with 128 schemas in context

Evidence

GPT-5.6 gained the most
- GPT-5.6-Sol and Terra each gained 10.6 points absolute over their own JSON baseline.
- Measured on the 309-entry BFCL v4 subset.
Older GPT models fell sharply
- GPT-4o, GPT-4.1 and GPT-5.4-mini were 19.7-26.9 points below JSON.
- GPT-4o went from 81.9 to 55.0.
Sonnet 5 went 80.8 to 96.2 on chaining
- Claude Opus 4.8 went from 80.8 to 94.2.
- GPT-4.1 collapsed from 98.1 to 40.4 (n=52).
GPT-5 went 71.9 to 96.9 on parallelism
- 13 of 14 models matched or beat JSON; the exception was GPT-4.1 (96.9 to 90.6).
- At N=48, tokens were 5,097 for JSON vs 3,535 for PTC.

How to Apply

Try PTC with a prompt today
- Pick one scenario that chains 2-3 tools and paste the prompt below.
- Compare correctness and turn count against your JSON calls.
Move big fan-out jobs to code execution
- Find tasks that issue dozens of parallel tool calls in one response.
- Build stubs, let the model use a loop or asyncio.gather, and run it in a subprocess.
Test multiline code output per model first
- If you use an older model, check whether generated code contains literal \n.
- If it causes syntax errors, keep JSON tool calling for that model.
Measure token cost by call count
- PTC has a fixed system-prompt overhead, so it costs more with few calls.
- Compare JSON and PTC tokens at your average call count.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: generates the script only, no real execution)

You are an agent that calls tools through a Python script, not JSON.
You may only import the typed stub functions below.

```python
{{paste your tool signatures as Python stubs}}
```

Rules:
- Write all needed calls in a single Python script.
- Handle independent calls with a loop or asyncio.gather.
- Print results with json.dumps.
- Use real newlines, never the literal characters \n.

Request: {{user request}}

Next step: run the same request with JSON function calling and compare call count, accuracy and tokens.

Terminology

Programmatic tool calling (PTC)The model writes one Python script that makes the tool calls instead of emitting JSON one call at a time. It is like running a macro instead of pressing remote buttons one by one.

BFCL v4A benchmark for function-calling accuracy. A fixed scorer checks whether the right function was called with the right arguments.

Fan-outIssuing N independent calls at once. Think of mailing a huge batch of letters in one go.

Context rotPerformance dropping because lots of irrelevant content fills the context. Like a junk-filled drawer where you can't find what you need.

Wilson confidence intervalA statistical way to show the uncertainty range of a proportion such as accuracy. The range widens with small samples.

Related Papers

Original Abstract (Expand)

Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.