How three Claude Code agents read the same rule across ten sessions and still didn't follow it — and the small script that finally fixed it.
TL;DR: We wrote a documentation-update rule into our agent instructions file. Three agents read it at the start of every session for ten sessions straight. None of them followed it — not once. The fix was a 60-line Python script that fires a warning on every tool call until the documentation catches up with the code. The rule went from zero compliance to full compliance in one afternoon.
Contents:
- The discovery — how we found out nobody was following the rule
- The rule itself and why it looked perfectly reasonable
- Ten sessions of quiet failure
- Why agents structurally cannot remember rules across long sessions
- The fork — rewriting the rule vs building a check
- What we built and what broke during rollout
- The principle — and what to do about it in your own setup
At Ayant-X we run multiple AI marketing agents — Claude Code sessions building and integrating an AI marketing platform for our agency. This is the story of what happened when three of those agents read the same rule across ten sessions, and none of them followed it.
The discovery
A routine documentation audit — one of the first we ran on the new multi-agent setup — turned up a problem. The integration spec for our backend audit module described a code structure that no longer existed. The module had been refactored twice since the spec was last touched. The spec was documenting code that had been deleted.
We went looking. The session logs showed every agent that had touched the file. Three sessions earlier, the spec was already stale. One of the agents had been working on the audit module and had encountered the gap. The session log captures the line, verbatim:
> "This is existing doc-debt, not from my change. Continuing."
The agent had read the rule about updating documentation at the start of the session. By the time it ran into the broken spec, the rule was three hours and forty thousand tokens behind it. So it categorized the staleness as someone else's problem, marked its task DONE, and closed out.
Three agents had looked at the same rule across ten sessions, and not one of them had followed it.
The instinctive first reaction was to call it agent laziness — after all, the rule was right there at the top of the file; every agent had read it at session start; any of them could quote it back when asked. Reading the rule again with fresh eyes surfaced a harder answer. We had written the rule ourselves, and we had asked the agents to do something that turned out to be structurally impossible across a fifty-thousand-token gap between when the rule was loaded and when it would have to fire.
The rule
The rule said: for every module, keep two documents — a developer guide and an integration spec. Update both when the code changes.
Five sentences of plain English, read by every agent at the start of every session because it sat under a numbered heading near the top of the agent instructions file. Three agents on the team, one rule, no good excuse for missing it.
The rule was not new — it had been there almost from the beginning of the project. It was not buried either: near the top of a short numbered list, with wording as direct as we knew how to make it. The argument that justified the original write-up was the obvious one: any agent that reads this file at the start of a session will see the rule, internalise it, and apply it during the work that follows.
That argument was correct as far as it went — the agents did read the rule, they acknowledged it, and they could quote it back when prompted. It was wrong about something else: it quietly assumed the agents would still be remembering the rule by the time they got around to finishing their tasks, which turned out not to be a safe assumption.
Most teams running AI agents already have a \`CLAUDE.md\` or an \`AGENTS.md\` or some local equivalent, and most of them are relying on that document to set agent behaviour the same way we were. The question this case study is really trying to answer is whether that kind of reliance survives contact with a working session that consumes 50,000 tokens of context before the agent ever reaches the moment when the rule matters.
Ten sessions, four modules
The rule went in early. All three agents picked it up on their next session start the way they always picked up the agent instructions document. Over the next ten sessions the rule kept being read at the top of every session, because the onboarding doc kept being read at the top of every session. Over that stretch the agents committed code across four modules — shipping features but never updating the matching integration specs alongside the code. The documentation drifted out of date quietly: no agent logged the violation, none of them flagged the gap in a handoff, and the session-closing documents that summarised each agent's work never once mentioned the rule the work was technically breaking.
The audit session was when we went deep. The integration spec, when we actually opened it, described a pipeline that had been replaced twice and referenced methods that no longer existed in the code. Of the four modules covered by the rule, three turned out to be partially stale and one was fully wrong; the fourth was clean only by accident, because that session's changes happened not to touch any of the code the spec described.
The blast radius was larger than the original violation. The engineer working from those specs had been writing automation against documentation that lied. We had shipped that automation. By the time the gap surfaced, downstream artifacts already depended on the wrong source of truth.
If we count it the way an engineer would: ten sessions across three agents is more than forty individual opportunities for some agent, at some point, to remember the rule on its own and update the matching documentation. None of them did. The gap only surfaced because an operator went looking through the specs by hand; not one of the agents ever flagged it themselves on the way out of a session.
Why agents fail
The agents were not being malicious, and they were not being lazy. They had genuinely read the rule at session start — we could grep for the read events in the transcripts and find them. What they could not do was act on the rule when the moment to act actually arrived, because by then the rule itself had drifted out of the part of their context they were paying attention to.
A Claude Code session runs inside a 200,000-token context window, with the agent instructions file sitting at the top. By the time the agent has read its task description, opened the related files, scrolled through the handoff notes from the previous session, and traced through the relevant code, it has already burned somewhere between thirty and fifty thousand tokens. The rule about updating documentation is still technically inside the window at that point, but operationally it is buried under the active work, with the agent's attention long since moved on. It is the kind of background fact that holds steady only when nothing else is competing for attention, and in a real session everything is competing for attention.
Compaction makes all of this worse. When Claude Code compacts a session to free up context, it summarises the work so far and drops most of the original input — and the rule was in that original input. Whether the compacted summary preserves it is essentially a coin flip, and any agent that crosses the compaction boundary in the middle of a task may well lose the rule entirely without realising it.
There is also no social pressure on the agent the way there is on a human engineer. When a human forgets to update a spec, code review eventually catches it — the pull request gets a comment, or the next standup asks whether the doc shipped, or someone reviewing the diff a week later raises the question. An AI agent working alone has none of those backstops: no reviewer noticing the gap in the PR, no standup conversation, no asynchronous comment thread the next morning. Once the rule has slipped out of active attention, there is no built-in mechanism for it to come back.
The number worth citing here comes from a 2026 Pixelmojo study on AI coding agent compliance: behaviours specified in \`CLAUDE.md\` get followed roughly 70 to 90 percent of the time, while behaviours enforced via \`PostToolUse\` hooks get followed essentially 100 percent of the time. In other words, the rule was always going to be broken at some non-zero rate. Our particular case happened to land the rate at zero — every one of the forty-plus opportunities across three agents and ten sessions ended up on the failure side of the distribution.
The bug was not in the agents — it was in the rule, or more precisely in our assumption about what kind of rule was reasonable. We had asked the agents to remember something across fifty thousand tokens of intervening work. The action the rule asked for was not hard in itself; it was simply almost impossible to remember at the moment the action needed to be taken.
What our team had run into has a name worth using: AI agent compliance — the gap between the rules you wrote into your agent's instruction file and the behaviors the agent actually performs once it is forty thousand tokens deep in a task. Software engineers know this as the difference between policy and enforcement, the distance every CI/CD pipeline closes by refusing to merge code that violates a linter. AI agent compliance is the same gap, in a place nobody has wired the linter yet. Once you name it, the fix stops being about better wording in the doc and starts being about which check fires the moment the rule is broken.
The fork: paper rules vs enforcement
After the audit discovery there were really two options on the table.
Option A was to add the documentation-update requirement to every task's completion criteria — essentially to push the rule into a different document and force the agent to re-encounter it at task-closing time. Cheaper to implement, no new infrastructure to maintain, no new code to debug.
Option B was to build an actual check: a small script that compares code modification times against documentation modification times, and that fires a warning the agent sees on the next tool use — and on every subsequent tool use — until the gap between code and docs is resolved.
Our first impulse was Option A. The argument for it is always seductive — we'll just add it to the completion checklist, the agent will see the requirement when it tries to close, the rule will fire at the right moment. The argument is also wrong about the same thing the original rule was wrong about. A checklist item is exactly as enforceable as a paragraph in the instructions file if no machine verifies it. The agent that did not check completion criteria against reality for the last ten sessions is not going to start checking them on session eleven. We had spent ten sessions watching paper rules fail in one location. Adding another paper rule in a new location would not change the failure mode.
We chose Option B. The honest framing of the decision was that we needed to stop relying on agent memory and start relying on the file system, which keeps modification timestamps regardless of whether anyone — agent or human — is paying attention to them.
The principle the fork crystallised is that a rule that lives only in a document is really a request, and a check wired into a hook is the actual rule. The document invites compliance; the hook compels it. Most teams reading this will have considered both options at some point, and most teams default to Option A because it is cheaper in the short run. The cost shows up later, in the form of the next ten sessions of quiet violation.
What we built
The check is two pieces of code working together. The first, a YAML config that maps code directories to the documents that should track them:
\`\`\`yaml audit: code: ["audit/partials/", "audit/assets/js/"] docs: ["audit/DEVELOPER_GUIDE.md"]
dashboard: code: ["dashboard/assets/js/", "dashboard/components/"] docs: ["dashboard/DEVELOPER_GUIDE.md", "dashboard/TIMUR_Dashboard_SPEC.md"] \`\`\`
Each module gets a section. Code globs map to the documents that should track them. Adding a new module is one section, two lists. The script itself does not know anything about any specific project — the project specificity lives in the YAML.
The Python script is the part that does the check. For each module in the config it finds the most recent code file by modification time, compares against the modification time of every required doc, and emits a warning when the code is newer. If a configured doc does not exist, it emits a hard fail. A recency flag scopes the check to modules touched in the last twenty-four hours — keeps the warnings about the current session's work, not older accumulated debt. Old debt is its own task in the backlog, not an interruption while you code.
Wiring it in took one line in our existing PostToolUse hook, which was already running on every \`Write\` and \`Edit\` operation — we just added the new call to the existing loop. Nothing else around it changed: no new process to launch, no configuration changes, and no onboarding cost for any new agent joining the team afterwards.
The warning itself is deliberately non-blocking — exit code zero, not two — so the agent keeps working. What it does instead is reappear on every subsequent tool call until the documentation gets touched: every \`Write\`, every \`Edit\`, every \`Bash\` invocation surfaces the same line again. The repetition is the actual mechanism. The agent can't forget the rule because the rule simply will not stop showing up until the situation is resolved.
What broke
Three implementation errors surfaced during rollout, and each one is worth describing specifically rather than collapsing into a generic "we ran into bugs" sentence.
The first config pointed at the wrong document. We mapped the backend audit module to a spec that had been the trigger for the whole investigation. By the time we shipped the check, the engineer had already created a replacement spec and the old file was no longer the live version. The script was checking a doc nobody was maintaining. The check fired correctly against the wrong target.
The second was a modification-time ambiguity from \`git pull\`. A pull operation on a machine that had been offline for a while set the modification time on two files within two milliseconds of each other. The script's comparison treated the code file as newer than the doc by a margin too small to be meaningful. It fired a false warning. The immediate fix was a manual \`touch\` on the doc and a follow-up question about whether the comparison should require a larger delta. The deeper lesson: filesystem timestamps are not commit timestamps. A check that depends on \`mtime\` will sometimes see \`git\`'s own bookkeeping as a real edit.
The third was a mapping to a document that did not exist. We mapped a module to a spec thinking the spec had been written. It had not been. The script fired a hard fail because the configured doc was missing. The fix was to remove the mapping until the spec was actually written. The lesson: a check that fires on a target that does not yet exist is louder than helpful.
Three errors in the rollout of a script whose entire purpose is to catch errors. The script itself did the job it was written for; what failed was the configuration around it, and the mapping between modules and their documentation needed several rounds of iteration before it actually reflected the state of the codebase.
The principle
A rule that lives in a document is doing one job — explaining why something matters — and a check that lives in a hook is doing a different job, answering whether the work has actually been done. These are not interchangeable. Documents are how a team teaches context; hooks are how a team enforces behaviour. Most of the failure mode we walked through above came from asking a document to do the enforcement job, which it cannot do reliably across the kind of distance an active session covers.
The generalisable rule we ended up writing into our own playbooks:
If a rule has been violated twice, automate it. The agent will not do better on the third attempt.
That is the practical core of AI agent enforcement: when the agent can't or won't enforce a rule on itself, enforcement has to come from the surrounding system instead. The same logic generalises well beyond documentation updates — it applies to commit message format, to required test runs before merge, to file size limits, to almost anything a team has written into a \`CLAUDE.md\` and then quietly watched the agents route around. If a team took the trouble to write a rule, it was because the rule mattered to someone; if it still matters and the agents are not following it, the rule needs to graduate from a sentence into a check.
There is a second principle that follows from the first. Every check should still come accompanied by its document. The document is what tells the agent why the check exists, and the check is what answers the operator's question of whether the agent actually complied. Together they form a working pair; alone, each one fails in its own characteristic way — the document fails because nothing enforces it, and the check fails because nobody understands why it is there.
We arrived at this lesson independently and so did most of the AI tooling industry through 2025 and 2026. A 2026 Zylos Research review found that every major agent runtime — Claude Code, OpenAI Codex CLI, LangChain, Google ADK, AutoGen, Semantic Kernel — independently converged on PostToolUse-style hooks as the answer to rule enforcement. We have not invented the pattern. We have a production case study where the rest of the industry has theory. That is a different kind of contribution, and the one we can actually offer.
What to do if you're running AI agents right now
If you have a \`CLAUDE.md\` or equivalent, audit it today. Pick three rules you care about most. Grep your last ten session transcripts for evidence that agents actually followed them — not that they read them, but that the behavior happened. If any rule shows a gap, that rule is a candidate for a hook.
Start with the cheapest check you can build. A PostToolUse hook that greps for a file, compares a timestamp, or counts lines in a test file. The check does not need to be smart. It needs to be present. A dumb check that fires every time beats a smart rule that fires never.
Do not remove the rule from the document when you add the hook. The document still teaches new agents why the rule exists. The hook just makes sure the teaching lands. One without the other is incomplete.
The pattern scales. We started with one documentation-update check. Within weeks it grew into a multi-gate enforcement layer covering commit format, test coverage, spec reviews, and task lifecycle. Each gate started the same way — a rule that got violated twice, then got automated. The compound effect is that agents working inside the system produce more consistent output than agents working from instructions alone, because the system remembers what the agents forget.
Three agents, one rule, ten sessions of quiet failure. The check took an afternoon to build. The rule is still in the instructions file, because agents still need to know why it exists. The difference is that now there is a script underneath it that does not care whether the agent remembers.
Related: Quality Gates for AI Agents — how the single hook grew into a full enforcement layer. 1922 Lines That Blocked 4 AI Agents — same pattern, different domain. We Had to Onboard a Human Into Our AI Team — what happens when two AI-driven setups merge. How our signal engine works — the same detect-before-it-costs principle applied to marketing. More from the series.