RL Does Exactly What You Reward. The Environment Decides What That Is.
Reinforcement learning (RL) environments are how AI labs now teach models to do jobs, not just answer questions. We built one for a real operational task: working out why an oil well's production dropped. Then we trained a small open model on it, from 0% to 88.7% correct on unseen cases. RL did its job precisely. Within ten steps it found every cheap route to a high score, and once the reward was right it polished the last few points. But what the model learned was decided before RL started, by the environment, the reward and how we wrote the worked examples. Here is what we built and four lessons that apply to any environment.
01 — the idea
An RL environment is a task, a sandbox and a grader
Fine-tuning shows a model the right answers. RL lets the model try, scores each attempt, and makes the attempts that scored well more likely. For that you need three things: a stream of tasks, a sandbox where the model can act, and a grader that scores the outcome with a program, not a person. Together they are an environment.
Ours copies a common job for a production engineer. A monitoring system flags a well: "oil down 18% in 30 days". The model gets read-only SQL access to a field database of 50 wells and a year of daily data, runs queries, and submits a root cause and an action. The grader knows the true cause because the generator injected it. The database never contains it.
02 — test the test first
Before training, prove the environment can't be gamed
There are eight possible causes, among them a failing pump, scale build-up, water breaking through, a mis-set choke valve, a drifting meter and plain natural decline. Each leaves its own signature across different tables. Before any model touched the environment, we checked two things with scripted players. Can the answer be found from the data? And can it be guessed without looking?
A hand-written set of engineering rules, using only SQL, got 95.5% right, so the signal is there. That figure is a little generous, since we tuned the order of the rules on errors from these same cases. Shortcuts that never read the evidence tables topped out at 35.7%. That last number was 50% at first. The size of the alert's drop gave the cause away, because faults dropped production more than natural decline did. We fixed it by drawing the drop first and then tuning each fault's severity to match. A model trained on the leaky version would have learned to read the alert, not the data.
03 — what happened
From 0% to 88.7%, and where the gain came from
We used Qwen3-1.7B, a small open model, and trained a LoRA adapter, a small add-on about 2% of the model's size that leaves the original weights untouched. Out of the box it scored between 0% and 3%. It almost never submitted an answer, and kept repeating queries that failed. RL learns nothing from a model that always fails, because there is no better attempt to reinforce. So we first fine-tuned it on worked examples written by the rule set from section 02.
Rewriting those examples took the model from 0% to 86%. RL, using GRPO, then added about four points. GRPO tries each case several times and favours the attempts that scored above the group's average.
The four lessons below cover what went wrong along the way, and what moved each step.
04 — lesson one
RL will find the cheapest way to score, so the reward is the spec
Our first reward penalised a wrong, costly intervention, such as replacing a pump that was fine. It did not penalise missing a real fault. That sounds reasonable, since sending a crew is expensive. RL disagreed. Within ten training steps the model had learned that answering "natural decline, no action" was a free hedge whenever it was unsure. Its answers of "scale" fell from 13 to 0. Accuracy dropped from about 50% to 42%.
We also found that always answering "choke change" earned a small positive reward. The fix was blunt: every wrong answer, and no answer, now costs the same. A test now checks that no constant answer can beat zero. Every reward has a cheapest route to a high score. Your job is to make sure that route is doing the task.
05 — lesson two
Worked examples must show the arithmetic
Our first examples came from the rule set, which did its calculations in code and wrote out only the conclusion. The model learned the order of checks perfectly, and reached 48%. But it could not tell water breakthrough from other faults: 2 right out of 14. It had never seen the comparison that decides it.
We changed the examples so that every check is written out with its numbers, the threshold and the verdict. Nothing else changed. Water breakthrough went from 2 to 11 out of 14 and overall accuracy from 48% to 69%. We had tried RL on the first model before this, and it got nowhere: RL can sharpen reasoning a model already writes, but it did not invent reasoning the model had never seen.
06 — lesson three
Make the model reason first and decide last
The reasoned examples created a new problem. Choke changes, which the first model got 12 out of 12, fell to 4. The cause was the first word of each turn. A turn that ended the investigation opened with "Checks:", and a turn that continued it opened with "Findings so far:". So the model had to choose between stopping and continuing on its very first word, before it had written any reasoning.
In the transcripts it would write the right sum, "choke 48/64 → 37/64, so a choke change", then carry on investigating and conclude something else. We gave every turn the same opening, "Findings:", and moved the decision ("Next:" or "Conclusion:") to the end. Choke changes went back to 12 out of 12, scale from 2 to 8 out of 8, and overall accuracy rose from 69% to 86%. The format of an example decides when the model commits, not only what it says.
07 — lesson four
A good model gives RL little to learn from
GRPO learns only from cases where some attempts succeed and others fail. That gap is the signal. Once the model was right 86% of the time, most cases went all right or all wrong. In each training step of 8 cases with 8 attempts each, only 0 to 5 cases had mixed results. The rest taught nothing.
On 256 fresh cases, RL moved accuracy from 84.8% to 88.7%. Compared case by case, it fixed 17 and broke 7. That is statistically significant, but only just (p ≈ 0.04), from one run, and we tested two checkpoints. Treat it as a small, real-looking gain, mostly on the faults that need a calculation, with fewer harmful interventions (3.5% down to 2.3%). It is not proof that RL did the heavy lifting. It didn't.
08 — what this does not show
Limits we want to be clear about
The field is synthetic. Decline curves and fault signatures use realistic ranges, but they are not fitted to real wells, and a production engineer has not yet reviewed them. This is a study of how to build and train on an environment, not a model ready to troubleshoot real wells.
The task is procedural. Each case has one fault, and a 60-line rule set solves 95.5%. Knowing which checks to run is most of the job. Harder versions need misleading evidence and more than one fault at once.
We have no comparison with a large model yet. We don't know how a frontier model would score on the same cases, so 88.7% shows progress for a 1.7B model, not a ceiling.
09 — the conclusion
Most of the work is in the environment and the examples
RL did exactly what we rewarded, loopholes first. Once the reward was right, it added four points. The environment and the worked examples decided everything else.
If you are building an environment for an operational task, we would spend the effort in this order. First, test the environment with scripted players until shortcuts fail and the expert succeeds. Then write a reward where no lazy strategy pays. Next, write worked examples that show every number and leave the decision to the end. Run RL last, to polish a model that already mostly works.
Next we will compare against a large model on the same cases, have a production engineer review the fault library, and add harder cases with misleading evidence.