Applications of Large Language Models to SQL Learning | AI Paper Digest

TL;DR Highlight

An o4-mini-based agent auto-generates SQL practice problems and grades student submissions at human-level accuracy.

Who Should Read

Developers building SQL education platforms or automated code grading pipelines. Backend engineers designing code evaluation systems with LLM-based agents.

Core Mechanics

Automatically synthesizes SQL practice problems + pedagogical metadata (difficulty, concept tags, etc.) based on real-world data cleaning scenarios
Using a multi-step 'operator planning → SQL generation' pipeline instead of a single prompt significantly improves reference SQL accuracy
OpenAI o4-mini achieves near-SOTA performance even in zero-shot (no examples) settings, rivaling supervised SOTA pipelines
Evaluates student-submitted SQL against a rubric, auto-generating partial scores and improvement feedback
Deployed in a real class — compared LLM grading against 6 human graders across 326 submissions for 4 exam questions, with competitive results on most question types

Evidence

Multi-step reference answer generator substantially outperforms single-prompt baseline, approaching SL-based SOTA
LLM grading is competitive with human graders across most question types in a real-class study with 326 submissions × 6 graders
o4-mini achieves SOTA-level performance in zero-shot settings without supervised training

How to Apply

When building a SQL education platform, call the LLM twice during problem generation — first to plan which operators (JOIN, GROUP BY, etc.) to use, then to write the SQL — and you'll get better quality than a single prompt.
When building an auto-grading pipeline, define a rubric with partial scoring criteria in JSON and let the LLM judge each criterion, giving you human-level grading feedback.
When automating coding assignments, separating the pipeline into three independent agents — problem synthesis, reference answer generation, and grading — lets you improve or swap each stage independently.

Code Example

snippet

# Multi-step SQL reference answer generation prompt example (based on o4-mini)

# Step 1: Operator planning
planning_prompt = """
Plan the list of operators needed to solve the following SQL problem.
Problem: {problem_description}
Table schema: {schema}

List the operators to use in order, and explain why each is needed.
JSON format: {"operators": [{"name": "JOIN", "reason": "..."}, ...]}
"""

# Step 2: SQL generation based on the plan
sql_gen_prompt = """
Write executable SQL based on the following operator plan.
Problem: {problem_description}
Schema: {schema}
Operator plan: {operator_plan}

Requirements:
- Use standard SQL syntax
- Reflect all planned operators
- Explain the purpose of each clause with comments
"""

# Step 3: Rubric-based grading
grading_prompt = """
Grade the student's SQL submission according to the rubric below.

Reference SQL: {reference_sql}
Student submission: {student_sql}

Rubric:
- Correct table JOIN (2 points)
- GROUP BY clause accuracy (2 points)  
- WHERE condition completeness (2 points)
- Column selection accuracy (2 points)
- Query executability (2 points)

Return the score and reason for each item, along with suggestions for improvement, as JSON.
{"scores": [{"criterion": "...", "score": N, "feedback": "...", "suggestion": "..."}]}
"""

Terminology

zero-shotSolving a problem directly without any examples. Like taking an exam without seeing sample questions first. The opposite is few-shot (showing a few examples before the task).

agentic workflowAn autonomous pipeline where an LLM executes multiple steps sequentially rather than just responding. Like AI doing the 'plan → execute → review' cycle that humans do.

rubricA grading guide that breaks down scoring criteria. Like 'if this item is correct, N points.' Giving a rubric to an LLM enables consistent partial-credit grading.

operator planningThe step before writing a SQL query where you plan which SQL operators (JOIN, GROUP BY, etc.) to use. Like drawing a blueprint before building.

o4-miniOpenAI's reasoning-specialized lightweight model. Strong at complex step-by-step thinking tasks with high performance-to-cost ratio.

SOTAState of the Art. The current highest-performing baseline in a given field — essentially 'best known performance.'

pedagogical metadataEducational metadata about a problem — difficulty, concepts covered, learning objectives, etc.

Related Papers

Original Abstract (Expand)

We present a Large Language Model (LLM)-assisted SQL learning system that closes the loop from problem discovery to grading. Grounded in real-world data-wrangling scenarios, our agentic workflow (i) synthesizes industry-style practice problems with pedagogical metadata, (ii) produces executable reference SQL via a multi-step operator-planning pipeline, and (iii) grades student submissions against rich rubrics while explaining partial credit and surfacing actionable feedback for revision. We evaluate two core capabilities. First, on a large corpus of realistic SQL problems, our zero-shot, multi-step reference-answer generator, implemented with OpenAI's o4-mini, substantially outperforms a single-prompt baseline while approaching the state-of-the-art pipelines trained with supervised learning. Second, in a classroom deployment, we compare LLM-assisted grading with human graders across four exam questions, encompassing 326 submissions evaluated by six graders. The results indicate that LLMs can provide grading signals competitive with those of human graders for many question types. Overall, the system is designed for responsible educational use through real-world problems, generated reference solutions, and grading assistance. Together, these features enable scalable practice generation and grading, which improves student learning while augmenting instructor capacity.