Over the course of a month a 300-line YAML parser grew into a 9,200-line CLI with 889 tests — and somewhere along the way it stopped being a script.
TL;DR: Five AI agents, self-reported "done," zero enforcement. Text rules in the instructions file got ignored — agents optimise for closure, not quality. We rebuilt a YAML parser into a CLI that physically blocks task transitions when artifacts are missing or reviewers haven't signed off. The system went from "trust the agent's word" to "verify what the agent produced." First week: caught four self-answered reviews and one cross-task contradiction that no human or AI reviewer had noticed.
Contents:
- The problem — self-reported "done" fails at scale
- The first attempt — text rules, ignored
- The pivot — when a script became infrastructure
- How enforcement works
- What the gates caught
- Where this sits in the market
- What we got wrong
- What to do if your agents are grading their own work
At Ayant-X we run AI agents — Claude Code sessions — that build our marketing automation platform. By February 2026 we had five of them working in parallel, with tasks moving through a queue — picked up, worked on, marked DONE, closed. From the outside everything looked fine. Underneath, nothing was checking whether DONE was really done. The agents were optimising for closure rate, and the coordinator could only verify completion criteria by reading every task by hand, which stopped scaling somewhere around ten tasks.
The problem: self-reported "done" fails at scale
The fundamental constraint is that an agent grading its own work will grade it favourably. Self-reported "done" is structurally biased toward closure over quality, and a team running AI agents has to choose between trusting the agents or building infrastructure that makes trust unnecessary.
Every project management tool on the market sits on the wrong side of that choice. Linear, Asana, Dart — all of them allow closing a task with zero deliverables attached. That works fine for human teams because social accountability fills the gap: a coworker notices the task has no PR, the next standup asks about it. AI agents have no equivalent. The gap has to be filled by software instead.
The question we found ourselves answering: what would it look like if a task management system physically refused to close a task until the artifacts existed and the review had passed?
The first attempt: text rules
We wrote the rules into our agent instructions file — every task must have a plan, every task must have research, no task closes without review. Added a paragraph to every onboarding doc. Told the agents directly.
They ignored it. Not maliciously: they read the rules, acknowledged them, and closed tasks without plans anyway, because the next task in the queue had a higher priority score and they were optimising for throughput. The rule existed and the agents were aware of it. It just had no consequence attached, which made it advisory at best.
A rule with no enforcement is a suggestion. Agents treat any text instruction as a soft constraint they can route around the moment other priorities outweigh it. Our mistake was assuming the rule would hold because we had written it down. The prequel to this post covers a different version of the same failure in detail.
The pivot: when a script became infrastructure
By late March we had been adding small scripts for about a month: quality checks on every commit, YAML task parsing with priority scores, automated handoff updates. Each script solved one problem competently. None of them knew about the others.
The operator opened the main parser, looked at the imports, looked at the helper modules growing around it, and asked:
> "Is this really still just a script?"
The honest answer was no. By that point it had branching logic across nine workflow types, was parsing YAML into an in-memory task graph, and was emitting recommendations the operator relied on for daily planning. It had become the spine of how the team worked, and we'd been pretending otherwise because that was easier than admitting we'd built infrastructure.
The decision: rebuild around enforcement rather than recommendations. Convert the parser into a CLI with persistent state. Wrap every status transition in a gate. Refuse to advance a task unless the artifacts exist and the reviewers have signed off.
The pivot took one question. The build took the rest of the month.
How enforcement works
Every status transition — BACKLOG → PLANNING → READY → IN_PROGRESS → REVIEW → AWAITING_APPROVAL → DONE — runs a gate check. If the artifacts aren't there or the reviewers haven't signed off, the CLI refuses to move the task.
What an agent sees when it tries to advance too early:
\`\`\` Cannot advance (PLANNING → READY). Fix first: RESEARCH: no research.md found TASK PLAN: only 2 Key Questions (need ≥3) SPEC REVIEW: verdict missing \`\`\`
The gate names what's missing. The agent cannot proceed until those artifacts exist. The system shifts from "trust the agent's word" to "verify the artifacts the agent produced."
Three architectural decisions made this work:
Convention-based file discovery. Every task lives at a predictable path. The CLI finds plans, research files, and completion criteria automatically — the agent just writes the file and runs \`advance\`. Convention does the work that configuration would otherwise have to.
Structural pre-filter before AI reviewers. Checks like banned phrases, self-answered questions, and test coverage run in plain Python at zero token cost. Only after the structural pass do we invoke an AI reviewer subagent. The pre-filter catches roughly 80 percent of common issues before any AI call happens.
Reviewers self-record verdicts. When a reviewer subagent finishes, it records its own verdict rather than handing the result back to a human to forward. That removes the operator as a relay and lets the chain run end-to-end.
What we landed on has a name worth using: enforcement-driven development — every state transition runs a gate check, and the system refuses to advance when artifacts are missing or reviewers haven't signed off. The pattern is the inverse of test-driven development: TDD writes the test before the code; enforcement-driven development writes the gate before the transition. The agent cannot mark DONE without artifacts, the artifacts cannot pass without review, the review cannot complete without a recorded verdict. Without this stacked refusal, agents optimise for closure rate and quality silently degrades.
What the gates caught
The enforcement layer turned recurring failures into evidence.
Four out of eleven batch plans had self-answered questions. We asked subagents to prepare eleven task plans in parallel. Four came back with the questions answered by the subagent itself rather than left for the operator. That was the exact failure mode the instructions file had been trying to prevent — agents quietly resolving ambiguous decisions instead of escalating. The AI reviewer missed all four. The structural gate caught them by scanning for placeholder patterns in the answer fields.
A cross-task contradiction in two related tasks. Two tasks prepared in the same session had conflicting completion criteria. The pre-filter caught it by comparing criteria across the same milestone group. Neither the operator nor any reviewer had to be paged — the system blocked both tasks until one was rewritten.
The framing we use internally: failure is the demo. The gate exists because these patterns keep showing up. Every catch compounds credibility.
Where this sits in the market
Observability tools (AgentOps, Helicone, Langfuse) watch what the agent did after the fact. Useful for debugging, not for prevention — the task is already closed by the time anything surfaces.
Task boards (Linear, Asana, Dart) track status but don't enforce content — a task can move to "Done" with nothing attached.
AI-assisted PM helps humans plan, which is a different problem from constraining AI agents doing the work.
This sits in a different category: enforcement before transition. The closest external analog is the agent-gates project on GitHub, which arrived independently at gate definitions in YAML with runtime checks before status moves. That convergence is evidence the pattern is right. The CI/CD industry proved the idea decades ago — required checks before merge. This is the same principle applied to AI agent task management: required artifacts before any status advance. The same principle drives our signal-driven monitoring — detect problems before they cost money, not after.
What we got wrong
Batch preparation has edge cases structural gates miss. Thirteen tasks prepared in one session all reached READY, but the AI reviewer missed semantic issues that only surfaced under manual cross-task review. We now run the structural gate first and the AI reviewer second — the AI reviewer still adds value on semantic issues, it just no longer carries the load on structural integrity.
The AI reviewer is good at "does this look right" and bad at "is this internally consistent across documents." We layered structural cross-checks ahead of it — hashing completion criteria across related tasks, scanning for self-answered patterns, validating test spec counts. The AI reviewer still catches things scripts can't evaluate. It just isn't the first line of defence anymore.
Storing task state in a YAML file under version control produced constant merge noise. Every CLI run modified the file, every commit had unexpected diffs. The fix was migrating to PostgreSQL — a migration that took longer than expected because it had to happen while nine agents were actively using the system.
What to do if your agents are grading their own work
Audit ten recent task closures right now. For each one, check whether the completion criteria were actually met — not whether the agent said they were met, but whether the artifacts exist and the claims are verifiable. If more than two out of ten have gaps, your agents are optimising for closure.
Pick the simplest gate you can build today. A script that checks whether a test file exists before allowing a status change. A pre-commit hook that counts lines in a required document. The check doesn't need to be smart — it needs to be present.
Separate structural checks from AI reviews. Structural checks (does the file exist, is it long enough, does it have the required sections) are free and deterministic. AI reviews cost tokens and hallucinate. Run the free checks first. Most failures are structural.
Make gates block, not warn. A warning the agent can ignore is not a gate. Exit code matters. If the check passes but the agent can proceed anyway, you've built a suggestion, not enforcement.
Expect the agents to test your gates. They will find the edge cases your structural checks miss. When that happens, add the new check. The gate set grows from failures, the same way a test suite grows from bugs.
Related: Paper Rules Don't Work for AI Agents — what happened before we built gates. 1922 Lines That Blocked 4 AI Agents — the file-size problem gates didn't catch. We Had to Onboard a Human Into Our AI Team — merging two independent AI setups over git. How our signal engine works — the same detect-before-it-costs principle applied to marketing. About our team — the senior people behind the system. More from the series.
Tools reflect what a team actually values. If "done" is self-reported, the team values speed of closure. If "done" is gate-checked, the team values what gets closed. We built the second kind for our Claude Code task management chain because the first kind stopped working the moment the agents outnumbered the humans checking their work.