If Your Agent Wrote the Test, Ignore the Green Build

Do not hide fixtures inside pytest helpers. Pytest may wrap the grader later. It must not replace the oracle files.

Human-written cases

{ "cases": [ { "id": "refund-partial-usd", "input": { "order_id": "ord_1001", "paid_cents": 5000, "refund_cents": 1200, "currency": "USD" }, "expect": { "status": "ok", "refunded_cents": 1200, "remaining_cents": 3800 } }, { "id": "refund-overpay-rejected", "input": { "order_id": "ord_1002", "paid_cents": 5000, "refund_cents": 5001, "currency": "USD" }, "expect": { "status": "rejected", "reason": "amount_exceeds_paid" } }, { "id": "refund-zero-rejected", "input": { "order_id": "ord_1003", "paid_cents": 5000, "refund_cents": 0, "currency": "USD" }, "expect": { "status": "rejected", "reason": "amount_not_positive" } } ] } 

Those numbers came from you, not the model. Guard them like production credentials. They are the only truth for this task.

Minimal handler to grade

Label this stub as an example, not production.

# src/handler.py def handle(payload: dict) -> dict: paid = int(payload["paid_cents"]) refund = int(payload["refund_cents"]) if refund <= 0: return {"status": "rejected", "reason": "amount_not_positive"} if refund > paid: return {"status": "rejected", "reason": "amount_exceeds_paid"} return { "status": "ok", "refunded_cents": refund, "remaining_cents": paid - refund, } 

An agent may replace this file later. It cannot replace oracle/cases.json now. That restriction is the entire method.

Grader

# tools/grade.py from __future__ import annotations import json import sys from pathlib import Path ROOT = Path(__file__).resolve().parents[1] ORACLE = ROOT / "oracle" / "cases.json" def load_cases() -> list[dict]: return json.loads(ORACLE.read_text())["cases"] def main() -> int: from src.handler import handle from oracle.invariants import check as check_invariants failures = [] for case in load_cases(): got = handle(case["input"]) try: check_invariants(got) except AssertionError as exc: failures.append( {"id": case["id"], "invariant": str(exc), "got": got} ) continue if got != case["expect"]: failures.append( {"id": case["id"], "expected": case["expect"], "got": got} ) if failures: print(json.dumps({"pass": False, "failures": failures}, indent=2)) return 1 print(json.dumps({"pass": True, "count": len(load_cases())})) return 0 if __name__ == "__main__": raise SystemExit(main()) 

# oracle/invariants.py def check(result: dict) -> None: if "remaining_cents" in result: assert result["remaining_cents"] >= 0, "remaining_cents is negative" if result.get("status") == "ok": assert "refunded_cents" in result, "ok result missing refunded_cents" 

Run it before the agent starts work. Run it after every candidate patch. A red grader is information you wanted.

export PYTHONPATH=. python tools/grade.py echo $? 

A rewritten oracle is contamination. Treat that diff as an incident. Do not chat the model through it.

Mechanical freeze

Policy comments do not stop an agent loop. Git status does stop an agent loop. Enforce the freeze with a command.

# tools/freeze_check.py from __future__ import annotations import subprocess import sys FROZEN = ( "oracle/", "tools/grade.py", "tools/freeze_check.py", ) def changed_files() -> list[str]: staged = subprocess.check_output( ["git", "diff", "--name-only", "--cached"], text=True, ) unstaged = subprocess.check_output( ["git", "diff", "--name-only"], text=True, ) names = set() for block in (staged, unstaged): names.update( line.strip() for line in block.splitlines() if line.strip() ) return sorted(names) def is_frozen(path: str) -> bool: return any(path == prefix or path.startswith(prefix) for prefix in FROZEN) def main() -> int: blocked = [path for path in changed_files() if is_frozen(path)] if blocked: print("frozen path edited:") for path in blocked: print(f"- {path}") return 2 print("freeze check passed") return 0 if __name__ == "__main__": raise SystemExit(main()) 

Wire one command for humans and CI:

python tools/freeze_check.py && PYTHONPATH=. python tools/grade.py 

If freeze check returns 2, reset frozen paths. Do not negotiate with the model. Restore the answer key immediately.

git checkout -- oracle tools/grade.py tools/freeze_check.py 

Decision table

Paste this table into the pull request.

If two rows apply, take the stricter row. Do not average conflicting rows. Strictness is the point of the table.

Cheap generation still needs a local grade

You do not need a paid API here. You need retries, a branch, and freeze. Grade locally after every generated hunk.

Disclosure: This article was prepared as part of MonkeyCode’s product outreach.

MonkeyCode is an open-source coding assistant. It provides free model access and a free server option. Point it at src/ only. Keep oracle/ on your machine. Grade every candidate patch locally.

A practical sequence looks like this:

git switch -c agent/refund-oracle PYTHONPATH=. python tools/grade.py > /tmp/grade.json # give the assistant /tmp/grade.json # do not grant write access to oracle/ git apply --check /tmp/src.patch git apply /tmp/src.patch python tools/freeze_check.py || git checkout -- oracle tools PYTHONPATH=. python tools/grade.py 

Feed the failing JSON into the assistant. Do not grant oracle write permission. Hidden fixtures are a feature, not a problem.

The free server is useful for patch generation. It is not useful as a grader host. Your laptop already knows the answer key.

Reproducible test plan

Run this plan on a clean clone. No extra services are required today.

1. Commit the three oracle cases above.
2. Replace src/handler.py with a constant ok response.
3. Run the grader and confirm exit code 1.
4. Confirm the JSON lists all three case ids.
5. Restore reject paths only, then grade again.
6. Confirm refund-partial-usd still fails loudly.
7. Restore remaining_cents math and expect exit 0.
8. Edit oracle remaining_cents to the wrong number.
9. Run freeze_check and confirm exit code 2.
10. Restore the oracle and confirm freeze_check exit 0.

If step 9 does not fail, your freeze is theater. Fix the path prefixes before inviting agents.

Add one pytest wrapper if your CI demands pytest. Keep it thin. The wrapper should call grade.py and assert exit code 0.

# tests/test_oracle_wiring.py import subprocess import sys from pathlib import Path ROOT = Path(__file__).resolve().parents[1] def test_grader_exits_zero(): result = subprocess.run( [sys.executable, "tools/grade.py"], cwd=ROOT, env={**dict(**{k: v for k, v in __import__("os").environ.items()}), "PYTHONPATH": str(ROOT)}, ) assert result.returncode == 0 

That test proves wiring. It does not prove the agent is honest. Honesty comes from the frozen fixtures.

Failure analysis

When the grader stays red, read the JSON. Do not open a long chat thread.

• One field differs: fix src/ and regrade.
• Every case returns one shape: input is ignored.
• An invariant fires: a written law was broken.
• Freeze check fires: the loop edited the key.

Teams excuse that last case constantly. Do not excuse it this time. Drop the entire patch.

A useful debug habit is boring. Save /tmp/grade.json per attempt. Diff those files, not chat logs. Chat logs are not replayable evidence.

mkdir -p /tmp/grades PYTHONPATH=. python tools/grade.py > /tmp/grades/$(date +%s).json 

If later attempts only change wording, stop. The model is negotiating. Your oracle already answered.

Limitations

Oracles are narrow on purpose. Equality is not product taste. This method will feel slow during spikes.

Do not use this approach when:

• you are still inventing the API shape
• expected output is honestly unknown
• the artifact is a plot, not a function
• there is no git history to freeze
• a human already watches every hunk live

An oracle can still be wrong. Change it in a dedicated commit. Never bundle an oracle edit with agent src/ edits.

The grader will not invent missing cases. You still add fixtures yourself. That work remains human by design.

This also fails for flaky time-based output. Freeze clocks in the handler boundary. Do not freeze live timestamps inside cases.

What this argument is not

This is not a model leaderboard post. This is not a latency or throughput claim. This is not a promise that free servers replace reviewers.

Cheap generation is useful in bounded loops. Self-graded generation is not quality control. Split those jobs in the repository.

Own the expected result in git. Let the loop struggle against that file. If the loop can edit the answer key, the green build is theater.

If you need a cheap src/-only patch generator against that gate, MonkeyCode’s free model access and free server option are enough to run the loop. Keep grading on your side of the freeze.