Teaching MiniCPM5 1B and 2B to be SQL agents on a MacBook

Early in this project, one of my evaluations went from 25% to 82.5% after 25 training steps. Then I looked at where its questions came from. They were drawn from the same 15 question templates and values the model was training on, and 23 of the 25 had appeared in training. That jump was mostly the model remembering questions it had already been rewarded for.

The model was MiniCPM5-1B, OpenBMB's one-billion-parameter model, and I was training it to answer questions about a database. It gets a question in plain English and a SQLite database it can query. It can run queries to look around, read the results or the error messages, and when it's ready it submits one final query, whose result is its answer. A model this size trains on my MacBook (an M3 Pro with 36 GB of memory) and runs in LM Studio from a 1.15 GB file, so if it can do the job, the whole thing fits on one laptop.

After that evaluation, I rebuilt the evaluation and the grader before trusting anything else. The grader turned out to have the same kind of problem, and most of this post is about how both were fixed and what the training did once the numbers could be trusted.

Where it ended up: on a test set of question types neither model trained on, the released MiniCPM5-1B gets 3.0% of its attempts right and my fine-tuned version 47.9%. The released MiniCPM5-2B, a model twice its size, gets 38.3%, and the same training took it to 73.1%. On Spider, a public benchmark whose databases neither model has seen, the gains are smaller: 6.6% to 29.2% for the 1B and 41.6% to 54.1% for the 2B, so there the released 2B is still ahead of the fine-tuned 1B. I'll explain what each test measures further down.

All the training and evaluation ran on the laptop. Small API models wrote and checked training questions for about $2 in total. The code, tests and engineering records are on GitHub.

The task

The model gets a question such as "Which customers have placed more than 10 orders? Return only their names." and two tools. run_sql runs a read-only query and shows up to ten rows of the result. submit_sql submits a final query and ends the attempt. The complete result of the submitted query is the answer, and anything the model writes in prose is ignored. A correct answer scores 1, a wrong one 0, and running out of turns or tokens without submitting scores -0.2.

Here's the released 1B on one of the test questions. I've cut two broken queries from the middle:

How many customers living in Paris have placed at least one order but never had an order with status 'shipped'?

1. run_sqlSELECT COUNT(*) FROM customers c WHERE c.city = 'Paris' AND (SELECT AVG(status) FROM order_items oi GROUP BY oi.order_id HAVING oi.status <> 'shipped') > 0
   errorno such column: status
2. run_sqlSELECT COUNT(*) FROM customers c WHERE c.city = 'Paris' AND NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id AND o.status = 'shipped')
   result0
3. replied in prose

   It appears that there are no records in the database to answer your question. Please verify the spelling of the city name and the status for each order. The query available to me is limited.

Never submitted, scored as no answer

It never calls submit_sql, so this attempt scores as no answer, and that's how most of the released 1B's attempts went.

Its last query is worth a closer look. It returned 0, and the correct answer on this database is also 0, but the query only checks that a customer has no shipped orders. It never checks that they placed an order at all, which the question asks for. It gets the right number here because every Paris customer in this database has at least one order. The grader also runs each answer on three other databases. One of them has far fewer orders, and 15 of its Paris customers have never ordered, so there the same query returns 16 where the answer is 1. On a different attempt the released model submitted almost exactly this query, and the grader rejected it for that reason.

After training, the 1B gets this question right on 10 of its 16 attempts. In this one it adds the missing condition as an EXISTS:

How many customers living in Paris have placed at least one order but never had an order with status 'shipped'?

1. run_sqlSELECT COUNT(*) FROM customers c WHERE c.city='Paris' AND EXISTS (SELECT 1 FROM orders o WHERE o.customer_id=c.id) AND NOT EXISTS (SELECT 1 FROM orders o2 WHERE o2.customer_id=c.id AND o2.status='shipped')
   result0
2. submit_sqlSELECT COUNT(*) FROM customers c WHERE c.city='Paris' AND EXISTS (SELECT 1 FROM orders o WHERE o.customer_id=c.id) AND NOT EXISTS (SELECT 1 FROM orders o2 WHERE o2.customer_id=c.id AND o2.status='shipped')
   result0

Matches the reference on all four databases

It still makes the original mistake sometimes. One of its six failures on this question is the query without the EXISTS, rejected on the same database.

A grader that's hard to fool

Most of the early work went into the verifier, the code that decides whether a submitted query is correct. Training rewards whatever the verifier accepts, so if it accepts wrong SQL, the model learns wrong SQL.

Each submission runs against four databases and has to match the reference query's result on all of them. Row order is ignored. Column order, duplicate rows, text and integer precision all count. Results over 10,000 rows return an error instead of being cut off, and queries that read the clock are rejected.

The four databases aren't random copies. One has very few orders, so plenty of customers have none, and that's the one that catches the Paris query above. Another is built to catch other kinds of almost-right SQL. It has two different customers with the same name in every city, two same-name customers who are both top spenders, prices exactly on the thresholds the questions use, dates outside the usual years, and cancelled orders big enough to change which category wins if you forget the status filter.

I tested the verifier with 21 plausible wrong rewrites of the reference queries. Eleven of them passed the previous version for at least one question. All 21 now fail for every parameter value, while correct rewrites (NOT EXISTS, LEFT JOIN ... IS NULL, BETWEEN) still pass.

Question type	Reference and rewrite	Old	New
Customers in a city	COUNT(*)COUNT(DISTINCT name)	16/16	0/16
Cheap products	price < Nprice <= N	18/18	0/18
Signups in a year	signup_date LIKE 'YYYY-%'signup_date >= 'YYYY-01-01'	5/15	0/15
Average price by category	AVG(price)AVG(DISTINCT price)	1/1	0/1
Frequent customers	SELECT c.nameSELECT DISTINCT c.name	4/4	0/4
Top spenders	GROUP BY c.idGROUP BY c.name	2/2	0/2

I regraded the archive from the first training run with the stricter checks, and 376 of the 2,266 answers that had been marked correct failed. Each of those had been rewarded during training.

What the scores measure

After the leaked evaluation, the evaluation questions moved to separate databases with different data, and each question type was assigned to one purpose.

Every shop question uses the same schema: an online shop with customers, orders, order lines and products. Training questions come from templates, which are a question with blanks, such as How many customers live in {city}?, plus the SQL that answers it. Filling the blanks with different values gives many questions of the same type. A "never-trained type" is a template the model never saw during training, so it has to write a new kind of query for a schema it knows.

Split	Used for
Trained types, new values: 16 questions	Trained templates with values and data held back from training
Selection set: 5 never-trained types, 19 questions	Picking checkpoints and stopping early
Check set: 5 other never-trained types, 18 questions	Scoring only at the start and end of a run
Test set: 9 never-trained types, 33 questions	Final results only

I wrote the test set last, after every design decision was made. It originally had ten types. Before any result existed, I wrote down a rule for excluding a test type if a generated training template asked nearly the same thing, and that rule excluded one type, the average order value in a country.

Each model's test set was scored once: one final evaluation run on the released model and one on the final checkpoint, and the results were never used to choose anything. Within that run, every question gets 16 attempts with normal random sampling, and the pass rate is the share of all attempts the grader accepted. So the fine-tuned 1B's 47.9% is 253 correct out of 528 attempts on 33 questions. With so few questions the range is wide: resampling which questions are in the set gives 36-59% for the fine-tuned 1B and 62-84% for the fine-tuned 2B.

Spider tests something different. It's a public text-to-SQL benchmark, and its dev set has 1,032 usable questions over 20 databases about concerts, car makers, dog kennels, tennis matches and more. Neither model trained on any of these schemas, so Spider measures whether the skills carry over to a database the model hasn't seen. Each Spider question got four attempts.

How the training works

The main method is reinforcement learning (RL), specifically group-relative policy optimisation (GRPO). For each question the model makes eight attempts and the verifier scores each one. Say three are correct (1 each), three are wrong (0) and two run out of turns (-0.2). The group average is 0.325. Each attempt's score minus that average decides what training does with it: the correct attempts get +0.675 and become more likely, the wrong ones get -0.325 and the two that ran out get -0.525, and those become less likely. A training step does this for 16 questions at once.

If all eight attempts are correct, every attempt scores exactly the average and the question teaches nothing. The same arithmetic also runs on a group where nothing is correct, which caused a problem I'll get to.

Training doesn't change the model's own weights. It trains LoRA adapters, small extra weight matrices added to the model's layers: 22.4 million weights for the 1B, about 2% of the model. That, together with Apple's MLX framework running everything on the MacBook's GPU, is what makes it fit on a laptop.

The first overnight run

The first long run started from the released 1B with a fresh adapter and ran 300 steps in 5.3 hours. Questions of the trained types were solved within about 30 steps. The never-trained types peaked at step 160 and then swung around:

• Never-trained types
• Trained types, new values

Each point on the never-trained line is only 44 attempts over three question types, so single points are noisy. Step 160 was also picked using this same evaluation, so its peak is optimistic. Some of the drops are too big to be noise, though: one question type went from 15 of 16 correct at step 160 to 0 of 16 at step 240.

The training logs show what changed alongside it. GRPO only learns from groups with mixed results. In the first 50 steps, an average of 9.8 of the 16 groups per step were mixed; in the last 100 steps, 1.1 were. By about step 60 the model was solving nearly every training question.

• Groups that can still teach the model

My reading is that after step 60 the model had little left to learn from its training questions, and the few updates that did happen moved its behaviour on other question types around without improving it. I didn't test that directly.

When more training made it worse

I then ran a series of short runs, all starting from step 160 of that run, and changed the code as each problem showed up. In these runs, when none of a question's eight attempts was correct, an API model (GLM-5.3-flash) wrote a solution, and if the verifier accepted it, it went into the group as an example. I call that the teacher below. These are the six problems, what I saw, and what changed:

One symptom I tracked through these runs is how unlikely the model finds the tokens it has just sampled. When that number rises, its outputs are getting more erratic.

• Reward and teacher not fixed
• Wrong answers no longer rewarded
• All six changes

In the red run, wrong SQL was still being rewarded and teacher solutions weren't filtered. Noise rose tenfold in 12 steps and accuracy on trained types fell from 98% to 63%. With the reward fixed (amber), it rose more slowly but still rose, and trained types still fell to 67%. A run with the teacher switched off, not shown, stayed between 0.03 and 0.06, which is why I think the teacher solutions were causing a good part of it. With all six changes (blue) it stayed below 0.16 over 20 steps and trained types held at 98-100%.

Because each run added changes to the one before, these runs show the changes working together. They don't show how much each one contributed on its own.

With everything in place, that 20-step run moved the check set from 40% to 59%. A longer run with twice as many questions per step then stalled: its selection score went from 41% to 45% in the first 10 steps and stayed there for the next two evaluations, so I stopped it.

Distillation got it past the plateau

RL can only build on answers the model sometimes gets right, so a question type it never solves gives it nothing. To get past the plateau I tried distillation, which here means plain supervised training on worked examples. GLM-5.3-flash wrote solutions for every training type, four parameter choices each, and only solutions the verifier accepted on all four databases were kept. That left 787 solutions across 212 types, which cost $0.12.

The model trained on those, mixed with 567 of its own correct answers so it wouldn't forget what it already knew: 86 updates at a low learning rate. The selection set went from 46% to 51% and the check set from 65% to 72%, with trained types unchanged at 97%. That was more than the stalled RL run had managed, so distillation before and after RL became the recipe for the final models.

Where the training questions came from

The question bank grew from 15 hand-written templates to 252 accepted generated ones. Claude Haiku 5.5 and GLM-5.3 propose templates, and a different model has to solve each one without seeing the reference answer. A template is rejected if its answer is the same on every database, if it depends on the order of tied rows, if one of its filters never changes the answer, or if its answers overlap a held-out question.

Reused a template name (now renamed automatically)136

Empty or NULL answer99

Solver could not answer it63

Reviewer rejected it61

Same answer on every database51

Malformed template38

Too many parameter combinations27

Placeholders did not match19

A filter never changes the answer14

Same as an evaluation question14

Duplicate of an accepted template10

Reference SQL fails9

Depends on tie order6

Overlaps a held-out question5

More than two thirds of the proposals were rejected. The most useful checks are the ones on the data, which the proposing model couldn't see. A template whose answer is empty, or identical on all four databases, would give free reward to SQL that returns nothing or a number copied from an earlier result.

When the model keeps failing a template, the generator writes simpler stepping-stone templates that teach its parts. The idea is the same as with mixed groups: a question the model can almost answer gives RL something to work with, and one it never gets right doesn't.

One pipeline for both models

The steps became one script that runs on any model:

1. Check
   Score the released model

   Every split, including the test set, 16 attempts per question
2. Train
   Distil from a fresh adapter

   787 verified teacher solutions, LoRA on all layers
3. Train
   GRPO with early stopping

   Up to six hours, checkpoints picked on the selection set
4. Train
   Distil again

   Teacher solutions plus the RL run's own correct transcripts
5. Check
   Keep the better checkpoint and score it once

   Whichever of RL or the second distillation scores higher on the selection set

Both models went through it unattended with the same inputs: the same 787 solutions for the first distillation, the same 202 generated templates when RL started, the same 884 solutions for the second distillation (the extra 97 are for templates modelled on Spider's training questions and rewritten for the shop schema), the same seed and the same six-hour RL budget. The 1B run took eight and a half hours and the 2B run just under 11.

• 1B
• 2B

Selection set

Check set

Trained types, new data

The first distillation pass did the most for both models. For the 1B, it also fixed the tool use: before it, 181 of 224 attempts in the in-run evaluation ended without a submitted query, and afterwards 6 did. The released 2B already submitted on most attempts, so it had less of this to fix.

RL added more for the 1B: 16 points on its selection set against 9 for the 2B. Its steps took about four and a half minutes against 10 to 36 for the 2B, so six hours covered 78 steps against 23. It also failed more often, and 65% of its groups had mixed results against 48% for the 2B.

• 1B
• 2B

Unlike the first overnight run, the 1B's selection score climbed for the whole run, with one small dip at step 50. The second distillation pass added a little on the never-trained sets for both models, and the pipeline kept it both times. For the 1B it cost some accuracy on trained types in the in-run evaluation (98% to 91%); the final scoring puts them at 94%.

Results on the shop questions

• Released
• Fine-tuned

0%25%50%75%100%

Test set9 never-trained types, 33 questions, scored once

1B

3.0% → 47.9%

2B

38.3% → 73.1%

Check set5 never-trained types, 18 questions

1B

10.1% → 74.0%

2B

73.6% → 96.2%

Selection set5 never-trained types, 19 questions

1B

8.9% → 58.2%

2B

47.4% → 76.0%

Trained types, new values and data16 questions

1B

12.5% → 94.1%

2B

69.5% → 98.8%

All attempts1,472 per model

1B

7.4% → 62.2%

2B

52.5% → 83.1%

On the shop questions, the fine-tuned 1B is ahead of the released 2B on every split except the check set, where the two are level. The selection-set numbers are a little flattering, because checkpoints were chosen on it. The 1B gained more points on the test set (45 against 35) because it started near zero, while the 2B removed a larger share of its mistakes: 56% of its test-set misses, against 46% for the 1B.

Most of the 1B's improvement is that it now submits an answer. The released 1B never submitted SQL on 625 of its 1,472 attempts and ran out of turns or tokens on another 624. The fine-tuned 1B submits on all but 30, so what's left is wrong SQL:

• Correct
• Wrong SQL
• Never submitted
• Ran out of turns or tokens

Released 1B

625624

Fine-tuned 1B

915527

Released 2B

773459192

Fine-tuned 2B

1,223212

Here is every never-trained question type for all four models. Read across a row to compare them:

0%100%

Question type	Released 1B	Fine-tuned 1B	Released 2B	Fine-tuned 2B
Test set
Latest signup in a country	22%	80%	86%	98%
City with the most orders in a country	0%	48%	25%	98%
Products above the average price	3%	61%	72%	97%
Distinct products bought in a city	0%	97%	94%	95%
Customers in a city with no shipped order	0%	67%	17%	80%
Customers who spent over an amount	0%	35%	12%	77%
Customers who bought from several categories	0%	21%	10%	69%
Busiest month in a year	0%	0%	6%	27%
Share of returned orders in a category	0%	0%	0%	5%
Average order value in a country (excluded)	0%	2%	17%	69%
Check set
Shipped revenue from a country in a year	0%	92%	41%	100%
Customers in a country who never ordered	2%	81%	78%	100%
Earliest order in a city	44%	88%	94%	100%
Orders per status in a year	0%	79%	92%	100%
Orders with more than N units	0%	17%	67%	77%
Selection set
Orders from a country	8%	97%	94%	100%
Signups from a country in a year	33%	100%	100%	100%
Best-selling category in a year	2%	56%	25%	98%
Repeat buyers in a category	0%	36%	11%	61%
Average order lines by status	0%	2%	2%	27%

Simple lookups, such as signups, orders from a country and distinct products bought in a city, are solved by both fine-tuned models. The biggest gaps between them are on questions that count groups passing a HAVING filter, where the 1B reaches 21-35% and the 2B 69-77%, and on picking the top item in each group: the best-selling category (56% against 98%) and the city with the most orders (48% against 98%). Three rows stay pale for every model: the busiest month, the returned-order share and average order lines by status.

The 1B does sometimes get the HAVING questions right, and it can recover from its own syntax errors on them. This is one of its 10 correct attempts out of 48 on the several-categories type:

How many customers have ordered products from at least 4 different categories?

1. run_sqlSELECT COUNT(*) FROM (SELECTcustomer_id FROM customers JOIN orders o ON o.customer_id=customers.id JOIN order_items oi ON oi.order_id=o.id JOIN products p ON p.id=oi.product_id GROUP BY customer_id HAVING COUNT(DISTINCT p.category)>=4)
   errornear "FROM": syntax error
2. run_sqlSELECT COUNT(*) FROM (SELECT o.customer_id FROM orders o JOIN order_items oi ON oi.order_id=o.id JOIN products p ON p.id=oi.product_id GROUP BY o.customer_id HAVING COUNT(DISTINCT p.category)>=4)
   result184
3. submit_sqlSELECT COUNT(*) FROM (SELECT o.customer_id FROM orders o JOIN order_items oi ON oi.order_id=o.id JOIN products p ON p.id=oi.product_id GROUP BY o.customer_id HAVING COUNT(DISTINCT p.category)>=4)
   result184

Matches the reference on all four databases

The pipeline 1B also beat the 1B from the development runs on every never-trained split, with less training: 47.9% against 44.7% on the test set, 74.0% against 70.1% on the check set and 58.2% against 50.0% on the selection set.

Results on Spider

Spider is where the 1B's limits show. The agent loop is the same as on the shop questions, with the Spider database's schema in the system prompt. An answer counts only when it matches the reference on the original database and on every one of Spider's test-suite variants, which are extra copies of each database with different data.

• Released
• Fine-tuned

0%25%50%75%100%

Pass rate per attempttest-suite grading

1B

6.6% → 29.2%

2B

41.6% → 54.1%

Solved in at least one of four attempts

1B

17.2% → 44.4%

2B

58.8% → 68.8%

With 1,032 questions the ranges are much tighter than on the shop test set: 27-32% for the fine-tuned 1B and 52-57% for the fine-tuned 2B. Both fine-tuned models score higher than their released versions on all 20 databases:

0%100%

Question type	Released 1B	Fine-tuned 1B	Released 2B	Fine-tuned 2B
Spider dev
Poker players (40)	14%	38%	61%	78%
Museum visits (18)	19%	44%	58%	76%
Employees and shops (38)	9%	51%	55%	75%
Students and pets (42)	2%	17%	58%	74%
Orchestras and conductors (40)	6%	47%	48%	72%
Voting contestants (15)	13%	53%	68%	70%
Singers and songs (30)	21%	38%	62%	68%
Document templates (84)	10%	32%	53%	65%
TV channels and shows (62)	6%	39%	50%	62%
Concerts and singers (45)	4%	34%	43%	62%
Dog kennels (82)	2%	22%	46%	60%
High-school friend network (56)	13%	45%	43%	59%
Courses and teachers (30)	11%	28%	46%	57%
Tennis players and matches (60)	4%	31%	40%	55%
Battles and ships (16)	8%	20%	34%	48%
Airlines and flights (80)	10%	26%	27%	44%
Student transcripts (78)	2%	24%	33%	40%
Real estate properties (4)	6%	25%	25%	38%
Car makers and models (92)	1%	20%	28%	32%
World countries and cities (120)	2%	13%	24%	30%

The 1B gains 23 points, but it's still below the released 2B on 18 of the 20 databases. Most of its failures are queries that don't run: 1,491 of its 2,717 wrong answers fail to execute, 947 of them with "no such column". In 864 of those, its last test query had just returned an error and it submitted anyway, and in 509 it submitted without running anything first. Only 146 of the fine-tuned 2B's 1,770 wrong answers fail to execute.

I haven't measured why. My guess is that the shop schema is small enough that the 1B rarely hit an execution error during training, so it never learned what to do after one.

Running it yourself

sqlrl/scripts/export_gguf.sh fuses a final adapter into its model and writes BF16, Q8_0 and Q4_K_M GGUF files. For the 1B, the Q8_0 file is 1.15 GB and the Q4_K_M 0.69 GB. No-think mode is the chat template default, because that's how the models were trained. At temperature 0 in llama.cpp, the 1B at Q8_0 gets 8 of the ten example prompts right and at Q4_K_M 6, so use Q8_0. docs/lmstudio.md has the settings, a system prompt and ten example questions with their measured success rates for testing it in LM Studio.

Everything else is in the repository: the training code, the verifier and its tests, the template generator, the dashboard, and the write-up, with the evidence for every number here in sqlrl/engineering/.

Next

The next stage continues both models from their final checkpoints on Spider's training databases, none of which is a dev schema. To stop a hard-coded answer from passing, each answer is graded on subsampled copies of its database. The stage also adds generated templates for the shop types both models still miss. Afterwards every shop split and Spider dev get scored again, to show what improved and what regressed. If the 1B still submits queries that have just failed after training on schemas where errors are common, my guess about the cause is probably wrong. The plan is in the repository.