Reinforcement Fine-Tuning: When a 4B Model Beats GPT-5.6

Reinforcement fine-tuning is post-training a model against a programmable grader instead of a fixed dataset of correct answers — and it just produced the most interesting number in LLM engineering. A 4B open model matched GPT-5.6 Sol on a retrieval task at roughly 100x less cost, reported in Neon’s write-up of a Castform post-training run. Same job, two orders of magnitude apart on the bill.

That result is not a claim that small models are secretly as good as frontier models. It’s a claim about where the money goes when you make a general model do a narrow job over and over — and reinforcement fine-tuning is the lever that gets it back.

TL;DR

RFT trains against a grader, not against answers. You write a function that scores a response; the model learns to score well. No labelled ideal outputs required.
The 100x gap is a tax refund, not a capability win. A frontier model re-derives your retrieval strategy on every call. Post-training bakes it into weights, so you stop paying at inference time.
A multi-turn search with gpt-5.6-sol runs >10s and ~$0.03 end to end. At a million queries that’s $30,000 and a UX problem.
Decompose the reward. The Castform setup grades retrieval, citation, and correctness separately — a single end-to-end score rewards lucky guesses from wrong evidence.
Data requirements are small. OpenAI recommends starting between several dozen and a few hundred examples, capped at 50,000.
Prompt first. RFT pays off on high-volume, gradeable, stable tasks. Break any one of those three and it’s a science project.

What reinforcement fine-tuning actually is

Mechanically, the loop is short: the model produces candidate responses, the grader scores each one, and the optimizer shifts weights so the high-scoring behavior becomes more likely. There is no labelled “ideal answer” anywhere in that loop — only a score.

That one substitution — grader in place of golden answers — changes which problems are trainable. Supervised fine-tuning needs you to write the ideal output for every example. Most real engineering tasks fail that test: you can tell whether a retrieval call returned the right document, but writing the perfect multi-turn search trajectory by hand for 500 queries is a job nobody will finish.

OpenAI’s RFT guide formalizes this with grader types you compose: string_check for exact matches, text_similarity, score_model for a 0–1 model-graded score, python for arbitrary sandboxed logic, and multi to combine several with weights. If you have ever written an eval, you have already written most of a reward function — which is the real reason this technique got accessible.

Why the 100x gap exists

Here is the mechanism, and it has nothing to do with the small model being smarter.

When you prompt a frontier model to do hybrid retrieval, every request pays for the model to work out how to search: which terms to pull, whether to run keyword or vector search, how to reconcile the two result sets, when to stop and answer. That reasoning is identical on call one and call one million. You are paying full frontier token rates to re-derive a fixed strategy, forever.

Post-training moves that strategy from the prompt into the weights. The Castform run trained the model to drive Neon’s Postgres search directly — lakebase_text for BM25, lakebase_vector for embedding similarity, rrf_merge for reciprocal-rank fusion — so the search policy is learned behavior, not per-call deliberation. The numbers Neon reports: a typical multi-turn search with gpt-5.6-sol takes over 10 seconds and about $0.03 end to end, while the 4B post-trained model does the same work for roughly 100x less.

Two things worth being precise about. First, $0.03 per query sounds trivial and isn’t: at a million queries it’s $30,000, and the 10-second latency is the part your users actually feel. Second, this is a narrow win. Ask that 4B model to do anything other than search your corpus and it will lose to the frontier model badly. That’s the trade, and it’s a good one only when the task really is narrow.

The same shape shows up elsewhere. Fireworks reports a 32B open model exceeding GPT-4o quality on customer-service function-call generation after SFT plus RFT, and a code-autofixer model built with Vercel running “10 to 40 times faster” than GPT-4o-mini and Gemini 2.5-Flash at matched quality. Different domains, same story: narrow task, learned policy, collapsed cost.

Reward design is the whole job

If you take one engineering lesson from the Castform run, make it this: decompose the reward.

Their reward function composes three checks:

Component	What it asks	Failure it catches
Retrieval	Was the correct source retrieved?	Search policy is wrong — bad query terms, wrong index
Citation	Was the right chunk cited?	Retrieved the doc, pointed at the wrong passage
Correctness	Was the final answer right?	Had the evidence, reasoned badly from it

A single end-to-end correctness score cannot tell those three apart. Worse, it actively rewards the model that produced the right answer from the wrong evidence — a hallucination that happened to land. Train on that signal long enough and you get a model that has learned to guess confidently, which is the exact behavior you started this project to eliminate.

This is also why teams with a real evaluation harness get to RFT faster than teams without one. If you’ve already built stage-level evals for your pipeline — the kind of decomposition I covered in the smevals write-up on LLM eval frameworks — your reward function is mostly a rename. If your only measurement is “does the answer look right,” you have weeks of work before any training run is meaningful.

The other half of the setup is throughput. RL post-training needs thousands of parallel rollouts, each one hitting the database, which is why the training loop lives next to autoscaling Postgres rather than a fixed-size instance. Your reward function’s latency becomes your training loop’s bottleneck — budget for it the way you’d budget for a hot path in production.

The decision: prompt, RAG, or post-train

Three conditions have to hold together before RFT beats prompting.

1. The task is stable. You’re optimizing a policy into weights. If the task definition changes monthly, you’re re-training monthly, and the frontier model’s flexibility is worth its price.

2. The task is gradeable programmatically. Not “a human can tell.” A function has to tell, cheaply, thousands of times per training step. Code compilation, retrieval hit/miss, schema validation, and exact-match all qualify. “Is this summary tasteful” does not.

3. The volume justifies it. Take your current spend, subtract the projected post-trained spend, and compare against training plus the engineering time to build the grader. At a few hundred calls a day, no arithmetic saves you. At a few hundred thousand, the case makes itself.

Note that none of these say “replace RAG.” The Castform model is doing retrieval — it just learned to drive the search rather than being told how on each call. If you’re still assembling the pipeline itself, start with the fundamentals in building a RAG pipeline from scratch and get chunking, indexing, and the retrieval layer validated first. Post-training a model to drive a retrieval layer you haven’t validated just teaches it to exploit your bugs — the same demo-to-production gap that swallows most agent projects.

Data volume is the pleasant surprise. OpenAI’s guidance is to start between several dozen and a few hundred examples, with limits of 50,000 training and 1,000 test examples, and an explicit note that dozens can be enough if quality is high. The caveat matters more than the number: your base model should already succeed on the task sometimes. RFT sharpens a capability the model has; it does not install one it lacks. If your base model scores zero, you need a bigger base model, not more RL.

Where this goes wrong

Grading the wrong thing. The reward function is the specification, and the model will satisfy it literally. A retrieval reward that only checks whether any result came back trains a model that always returns something.

Skipping the held-out set. RL will happily overfit to your grader’s blind spots. Keep a test set the training loop never sees and treat a widening gap between training reward and held-out score as a stop signal.

Assuming the frontier baseline is static. You are optimizing against a moving target. The cost of the model you’re beating drops every few months on its own, which shortens the payback window on your training investment — one more reason volume has to be real, not projected.

Confusing “small model” with “local model.” These are separate decisions. Running a 4B model on your own hardware is a deployment choice with its own tradeoffs, covered separately in running large models on small GPUs. A post-trained 4B model served from a managed endpoint is still a 100x cost win.

Training when the real problem is the pipeline. If retrieval quality is bad because the chunking is bad, RFT will teach the model to work around bad chunks. Fix the pipeline; then decide whether the remaining gap is worth a training run.

What to do this week

If you have one high-volume, narrow LLM task in production:

Instrument it. Log queries, retrieved context, outputs, and per-call cost for a week. You need the volume number before anything else.
Write the grader. Not the trainer — the grader. Score your existing production traffic with it. If you can’t write it, you’ve learned the answer already.
Decompose it into stage-level components the way the retrieval/citation/correctness split does.
Run the arithmetic. Current monthly spend versus post-trained spend versus the cost of building and maintaining the whole thing.

The pattern worth internalizing: the frontier model is a fantastic way to discover the right policy for a task, and an expensive way to run it a million times. Reinforcement fine-tuning is the migration between those two states — and as work on training against best-of-k sampling keeps showing, the technique is getting cheaper faster than most teams’ intuitions are updating.

FAQ

What is reinforcement fine-tuning (RFT)? Reinforcement fine-tuning trains a model against a programmable grader instead of a fixed set of correct answers. The model generates candidate responses, a reward function scores each one, and the weights shift so high-scoring behavior becomes more likely. OpenAI’s guide describes it as adapting a reasoning model using custom feedback signals rather than labelled outputs, which is what makes it viable when you can check an answer but can’t hand-write the ideal one.

How can a 4B model beat a frontier model like GPT-5.6 Sol? It doesn’t beat it in general — it beats it on one task. A frontier model has to infer your retrieval strategy from a prompt on every single call. A post-trained 4B model has that strategy baked into its weights, so it stops paying the reasoning tax at inference time. Neon’s write-up reports a multi-turn search with gpt-5.6-sol taking over 10 seconds and costing about $0.03 end to end, against roughly 100x less for the small post-trained model.

When should I use RFT instead of prompting or RAG? Use prompting first, always. RFT earns its keep when the same narrow decision runs at high volume, you can grade an answer programmatically, and inference cost or latency is a real constraint. If your task changes weekly, if you can’t write a grader, or if you serve a few hundred calls a day, prompting a frontier model is cheaper than any training run you could justify.

How much training data does reinforcement fine-tuning need? Far less than supervised fine-tuning. OpenAI’s guidance is to start with several dozen to a few hundred examples before investing further, with a hard ceiling of 50,000 training examples and 1,000 test examples. The documentation stresses that dozens of examples can be meaningful as long as they’re high quality — and that your base model should already succeed on the task sometimes, since RFT sharpens existing capability rather than installing new capability.

What makes a good reward function for a retrieval task? Decompose it instead of grading the final answer alone. The Castform/Neon setup splits reward into three checks — was the correct source retrieved, was the right chunk cited, and was the final answer correct — so the model gets signal about which stage failed. A single end-to-end correctness score rewards a model that guesses the right answer from the wrong evidence, which is exactly the failure mode you’re trying to train out.

Does RFT lock me into one model or vendor? Partly, and you should plan for it. OpenAI’s RFT is currently limited to o-series reasoning models, specifically o4-mini-2025-04-16. Open-weight paths are broader — Fireworks supports Llama, Phi 3/4, Qwen 2.5/3, DeepSeek V3 and R1 — and leave the weights in your hands. The portable asset in either case is your grader and your evaluation set, not the checkpoint; keep those in version control and a base-model swap becomes a retraining run rather than a rewrite.

Sources

Neon, “How Castform + Neon beats frontier models on price and efficiency” — the 4B model result, the ~100x cost figure, the >10s / ~$0.03 gpt-5.6-sol baseline, the lakebase_text / lakebase_vector / rrf_merge search stack, and the three-part reward function.
OpenAI, Reinforcement fine-tuning guide — the grader-based definition, the grader types, the o4-mini model restriction, and the dataset-size guidance (dozens to a few hundred to start; 50,000 training / 1,000 test maximum).
Fireworks AI, “Reinforcement Fine Tuning: Train expert open models to surpass closed frontier models” — the supported open base models, the 32B customer-service result against GPT-4o, and the Vercel autofixer’s 10–40x latency advantage over GPT-4o-mini and Gemini 2.5-Flash.

Written for umesh-malik.com — no-fluff technical writing on AI, Web Dev, and Engineering.