Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Who Should Read

Backend developer building a text-to-SQL feature who must decide which pipeline modules are worth the extra LLM cost

Core Mechanics

• Execution-feedback refinement always helps
  • R1 runs the SQL and asks the model to rewrite it only on an error or empty result.
  • It gained +1.83 to +4.89 pp on all four backbones, with CPP below 0.30 USD/pp everywhere.
  • Because it fires only on failures, average overhead stays small.
• Other modules depend on the model
  • Schema linking helps most on the weakest model (GPT-4o-mini) and fades on stronger ones.
  • CoT generation gave +4.24 pp on GPT-4o-mini but -0.39 pp on reasoning model DeepSeek-V4-Flash.
  • On the three reasoning models tested (DeepSeek-V4-Flash, Qwen3-235B, o3-mini), CoT and decomposition gave no consistent gain.
• Sampling five candidates is costly and capped
  • Selecting among five candidates cannot find an answer missing from the pool.
  • The best selector still trailed the oracle ceiling by 5.74 pp on GPT-4o-mini.
  • The paper therefore excludes candidate selection from its cost-efficient guideline.
• Input tokens follow structure, output follows the model

Evidence

• Effect of execution-feedback refinement
  • On BIRD dev, R1 raised GPT-4o-mini from 49.74 to 54.63 and Gemini-2.5-Flash from 59.84 to 62.32 (vs. N=1 baseline).
  • GPT-5.4 went from 61.28 to 63.30, with cost from 7.305 to 7.905 USD.
• Gemini stack vs. lean GPT-5.4
  • On BIRD dev, the Gemini-2.5-Flash stack scored 67.14% EX at 2.916 USD.
  • Lean GPT-5.4 (N=1) scored 61.28% at 7.305 USD.
  • The GPT-5.4 stack scored 69.43% at 16.237 USD.
• Transfer to Spider
  • BIRD-derived stacks applied unchanged to Spider dev gained +3.68 to +6.77 pp on all four models.
  • The GPT-4o-mini stack scored 80.17% at 0.315 USD, close to lean GPT-5.4 (80.27%, 1.879 USD).
• Guideline on five new models
  • On BIRD, T2 gained +1.24 to +2.61 pp over T1 and T3 gained +3.98 to +7.63 pp.
  • Claude Sonnet 4.5 went from 62.71 (T1) to 70.34 (T3), with cost from 8.62 to 18.94 USD.
  • Llama-3.1-8B-Instruct went from 37.61 to 44.07, with cost from 0.05 to 0.09 USD.

How to Apply

• Add rewrite only on SQL execution failure
  • Run each generated SQL first on a read-only database.
  • Only on an error or empty result, send the error message to the model to rewrite the SQL.
  • Return successful queries as they are to avoid extra calls.
• Drop CoT steps for reasoning models
• Do not add 'think step by step' prompts for reasoning models (o3-mini, Qwen3 thinking, etc.) and compare against direct generation.
  • For non-reasoning models like GPT-4o-mini, enable CoT generation and measure the difference.
• Try a stacked mid-tier model before upgrading
• Before upgrading the model, add masked-question example retrieval, pseudo-SQL schema linking, and R1 to a mid-tier model.
  • Record accuracy and total cost together on the same eval set.
  • Add five-candidate selection only when accuracy matters more than cost.
• Ship with safety guards
  • Run generated SQL only under a read-only account and in a sandbox.

Try It

# Paste into Claude or ChatGPT. A simplified version of the paper's R1 (rewrite SQL only on execution error)

You are a SQL fixing assistant. The SQL below had a problem when executed. Using the schema and question, fix the SQL.

[Schema]
{{스키마}}

[Question]
{{질문}}

[Executed SQL]
{{SQL}}

[Execution result]
{{error message or '0 rows returned'}}

Instruction: explain the cause in one line, then output only the fixed SQL in a code block.

Next step: in your app, execute the SQL first and call this prompt only on an error or 0 rows.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware