We handed an AI coding agent our entire playbook. Every doc. Every convention. All the context files that the team curated to let AI become aware about hidden rules of schema & code and the domain context.
Then we gave a second agent almost nothing.
The one with nothing did better.
That stopped me cold. So we dug in. Here is what we found.
The short answer: more context made the agent more confident, not more correct, and two sentences fixed what dozens of files could not.
Does more context actually improve AI coding agent accuracy?
Everyone building with AI agents now does context engineering, whether they call it that or not. CLAUDE.md. Cursor rules. “Here is how we do things around here.” The belief underneath is simple. More context, better output.
I wanted to know if that is true. Not as a feeling. As a number.
So we built a small evaluation (eval) harness for AI-enabled development and tested it on our own code. The goal was simple.
- Does a change in CLAUDE.md really help or does it make things worse?
- Should we add domain context files for AI to do better code generation? If yes, at what granularity level?
- Should we add context files for hidden schema rules?
How we tested it
We picked one real task from our codebase. A backend reporting routine. The kind of database procedure that pulls a pile of numbers together for a report. Not a toy. Real logic.
Then we deleted the working version.
We wrote 14 hidden tests. Each one checks a single thing the rebuilt routine has to get right. The agent never sees these tests. It only sees the task.
Then we ran a headless coding agent to rebuild the routine from scratch. Nine times. Three runs each, under three setups.
| Setup | What the agent was given |
|---|---|
| No context | The task. Nothing else. No house rules, no schema notes. |
| Full context | Everything. House style rules, a per-table schema reference, a map of how the routines call each other, even our own “gotchas” doc. |
| Full context + 2 lines | The exact same pile, plus two extra sentences in the main context file. |
The third setup is the second one with two lines added. One variable changed. Nothing else.
What happened
Here is the scoreboard. Each cell is the number of hidden checks that passed, out of 14, across the three runs.
| Setup | Checks passed (3 runs) | How it failed |
|---|---|---|
| No context | 0/14 · 13/14 · 13/14 | guessed a column name once; otherwise nearly perfect |
| Full context | 2/14 · 0/14 · 2/14 | crashed on a database error, or invented columns |
| Full context + 2 lines | 13/14 · 13/14 · 13/14 | one cosmetic miss, every time |
Look at the middle row. The agent with our full playbook was the only one that could not get the routine to run.
It failed in two ways that still make me smile.
First, it referenced the same temporary table twice inside one query. In our database engine, that throws a hard error. The query cannot even start. Two of its three runs died right there.
Second, it used column names that do not exist. It did not check. It just assumed they were there.
Now here is the part that gets me. We gave that agent a full schema reference. One page per table, every real column purpose, possible values explained. The reason is, the legacy software has many conventions that need reverse engineering deeply to code level to make sense of the value: 3 actually means “indirect assignments.” It had the answer in its hand. It still made the column up.
The no-context agent had no schema reference. So it opened the actual database and read the real columns before using them.
Read that again. The schema doc made the agent less likely to check the schema.
The grader that got fooled
We graded every run two ways.
One was the hidden tests. They actually run the code and check the output.
The other was an AI judge. It scores how closely the code follows our house style and matches the rest of our codebase. Think of it as a reviewer rating the code on looks.
The judge could not tell the setups apart.
| Setup | Hidden tests (does it run and is it right) | Style judge (does it look right) |
|---|---|---|
| No context | intermittent, up to 13 of 14 | ~0.9 |
| Full context | 2 of 14 | ~0.9 |
| Full context + 2 lines | 13 of 14 | ~0.9 |
Same style score. Wildly different reality.
The full-context run that crashed on the temp-table error still scored about 0.75 on structure. It had the copyright header. It had the right shape. It read like our code. It just did not run.
If we had trusted the looks-good judge alone, we would have shipped the broken one and called it a win.
That is the case for tests that execute. Looking right and being right are not the same thing, and only one of them runs in production.
The twist inside the twist
The full-context agent was not just wrong. It was wrong efficiently.
| Setup | Steps per run | What that shows |
|---|---|---|
| No context | 18, 49, 79 | poking around, inconsistent, sometimes flailing |
| Full context | ~40 every run | confident, direct, consistent |
| Full context + 2 lines | ~40 every run | confident, consistent, and now correct |
The context made the agent faster and steadier. Fewer steps. Less poking. It moved like it knew the place.
It just moved straight into a wall.
The no-context agent looked worse while working. It wandered. One run took 79 steps. But that wandering was it checking things it could not assume. The wandering is what saved it.
Why more context makes AI agents fail confidently
Picture two new engineers.
One read every page of your wiki before lunch on day one. Confident. Fluent. Talks like they have been here for years.
The other read nothing. So they keep opening the real code to check things before they touch anything.
Now hand both a job with a hidden landmine. The kind every team has. The thing that is not in the docs, because the people who know it got burned once and never wrote it down.
The confident one steps right on it. The cautious one pokes around and walks past it.
Our context turned the AI into the confident new hire. And there is a deeper reason it stepped on the mine. The docs taught it our real structure, which is faithful and complex. That faithful structure is exactly the one that reuses the temp table. The no-context agent never learned the fancy structure, so it wrote something flatter and plainer that never set the trap in the first place.
Context taught the shape of the work. It did not teach the scars.
The two lines that fixed it
We did not rewrite the docs. We added two lines.
One said: check column names against the schema before you use them. The other said: never reference the same temporary table twice in one query.
That is it. Two sentences naming the two traps.
The worst setup became the best. The temp-table crashes vanished. The made-up columns vanished. All three runs landed at 13 of 14, every time, tripped up only by the same tiny cosmetic detail, a 0 where the original returned an empty value.
A folder full of context lost to two lines of scar tissue.
And the asymmetry is the whole point. The full context was dozens of files and thousands of lines. The fix was 24 words. The 24 words won.
What I took from it
More context is not free. It is not automatically good.
Context that teaches structure but not traps just makes your agent faster at being confidently wrong.
We even handed it a gotchas doc. It just did not happen to list these two gotchas. So they may as well not have existed.
The most valuable thing in a context file may not be your architecture. Your agent can often infer that on its own, or go read it. It is the stuff nobody wrote down. The landmines. The “we never do X, because it quietly breaks Y.”
That knowledge lives in your senior people’s heads. This experiment told us to get it out of their heads and into the file.
I am still early on the real rule here. When does more context help, and when does it just make the mistakes more sure of themselves?
So I will ask you. If you use AI coding tools, how do you decide what goes into your context file? Have you actually checked whether it helps, or is it still a guess? And have you ever seen one tiny change move the result more than a big pile of effort did?
I would genuinely like to know what you are seeing.
