{"id":5007,"date":"2026-09-08T00:54:31","date_gmt":"2026-09-08T03:54:31","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/09\/08\/if-your-agent-wrote-the-test-ignore-the-green-build\/"},"modified":"2026-09-08T00:54:31","modified_gmt":"2026-09-08T03:54:31","slug":"if-your-agent-wrote-the-test-ignore-the-green-build","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/09\/08\/if-your-agent-wrote-the-test-ignore-the-green-build\/","title":{"rendered":"If Your Agent Wrote the Test, Ignore the Green Build"},"content":{"rendered":"
\n
<\/div>\n

Do not hide fixtures inside pytest helpers. Pytest may wrap the grader later. It must not replace the oracle files.<\/p>\n

<\/a> Human-written cases <\/h3>\n
\n
{<\/span> <\/span>\"cases\"<\/span>:<\/span> <\/span>[<\/span> <\/span>{<\/span> <\/span>\"id\"<\/span>:<\/span> <\/span>\"refund-partial-usd\"<\/span>,<\/span> <\/span>\"input\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"order_id\"<\/span>:<\/span> <\/span>\"ord_1001\"<\/span>,<\/span> <\/span>\"paid_cents\"<\/span>:<\/span> <\/span>5000<\/span>,<\/span> <\/span>\"refund_cents\"<\/span>:<\/span> <\/span>1200<\/span>,<\/span> <\/span>\"currency\"<\/span>:<\/span> <\/span>\"USD\"<\/span> <\/span>},<\/span> <\/span>\"expect\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"status\"<\/span>:<\/span> <\/span>\"ok\"<\/span>,<\/span> <\/span>\"refunded_cents\"<\/span>:<\/span> <\/span>1200<\/span>,<\/span> <\/span>\"remaining_cents\"<\/span>:<\/span> <\/span>3800<\/span> <\/span>}<\/span> <\/span>},<\/span> <\/span>{<\/span> <\/span>\"id\"<\/span>:<\/span> <\/span>\"refund-overpay-rejected\"<\/span>,<\/span> <\/span>\"input\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"order_id\"<\/span>:<\/span> <\/span>\"ord_1002\"<\/span>,<\/span> <\/span>\"paid_cents\"<\/span>:<\/span> <\/span>5000<\/span>,<\/span> <\/span>\"refund_cents\"<\/span>:<\/span> <\/span>5001<\/span>,<\/span> <\/span>\"currency\"<\/span>:<\/span> <\/span>\"USD\"<\/span> <\/span>},<\/span> <\/span>\"expect\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"status\"<\/span>:<\/span> <\/span>\"rejected\"<\/span>,<\/span> <\/span>\"reason\"<\/span>:<\/span> <\/span>\"amount_exceeds_paid\"<\/span> <\/span>}<\/span> <\/span>},<\/span> <\/span>{<\/span> <\/span>\"id\"<\/span>:<\/span> <\/span>\"refund-zero-rejected\"<\/span>,<\/span> <\/span>\"input\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"order_id\"<\/span>:<\/span> <\/span>\"ord_1003\"<\/span>,<\/span> <\/span>\"paid_cents\"<\/span>:<\/span> <\/span>5000<\/span>,<\/span> <\/span>\"refund_cents\"<\/span>:<\/span> <\/span>0<\/span>,<\/span> <\/span>\"currency\"<\/span>:<\/span> <\/span>\"USD\"<\/span> <\/span>},<\/span> <\/span>\"expect\"<\/span>:<\/span> <\/span>{<\/span> <\/span>\"status\"<\/span>:<\/span> <\/span>\"rejected\"<\/span>,<\/span> <\/span>\"reason\"<\/span>:<\/span> <\/span>\"amount_not_positive\"<\/span> <\/span>}<\/span> <\/span>}<\/span> <\/span>]<\/span> <\/span>}<\/span> <\/span><\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Those numbers came from you, not the model. Guard them like production credentials. They are the only truth for this task.<\/p>\n
  <\/a> Minimal handler to grade <\/h3>\n
Label this stub as an example, not production. <\/p>\n
\n
# src\/handler.py <\/span>def<\/span> handle<\/span>(<\/span>payload<\/span>:<\/span> dict<\/span>)<\/span> -><\/span> dict<\/span>:<\/span> paid<\/span> =<\/span> int<\/span>(<\/span>payload<\/span>[<\/span>\"<\/span>paid_cents<\/span>\"<\/span>])<\/span> refund<\/span> =<\/span> int<\/span>(<\/span>payload<\/span>[<\/span>\"<\/span>refund_cents<\/span>\"<\/span>])<\/span> if<\/span> refund<\/span> <=<\/span> 0<\/span>:<\/span> return<\/span> {<\/span>\"<\/span>status<\/span>\"<\/span>:<\/span> \"<\/span>rejected<\/span>\"<\/span>,<\/span> \"<\/span>reason<\/span>\"<\/span>:<\/span> \"<\/span>amount_not_positive<\/span>\"<\/span>}<\/span> if<\/span> refund<\/span> ><\/span> paid<\/span>:<\/span> return<\/span> {<\/span>\"<\/span>status<\/span>\"<\/span>:<\/span> \"<\/span>rejected<\/span>\"<\/span>,<\/span> \"<\/span>reason<\/span>\"<\/span>:<\/span> \"<\/span>amount_exceeds_paid<\/span>\"<\/span>}<\/span> return<\/span> {<\/span> \"<\/span>status<\/span>\"<\/span>:<\/span> \"<\/span>ok<\/span>\"<\/span>,<\/span> \"<\/span>refunded_cents<\/span>\"<\/span>:<\/span> refund<\/span>,<\/span> \"<\/span>remaining_cents<\/span>\"<\/span>:<\/span> paid<\/span> -<\/span> refund<\/span>,<\/span> }<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
An agent may replace this file later. It cannot replace oracle\/cases.json<\/code> now. That restriction is the entire method.<\/p>\n
  <\/a> Grader <\/h3>\n
\n
# tools\/grade.py <\/span>from<\/span> __future__<\/span> import<\/span> annotations<\/span> import<\/span> json<\/span> import<\/span> sys<\/span> from<\/span> pathlib<\/span> import<\/span> Path<\/span> ROOT<\/span> =<\/span> Path<\/span>(<\/span>__file__<\/span>).<\/span>resolve<\/span>().<\/span>parents<\/span>[<\/span>1<\/span>]<\/span> ORACLE<\/span> =<\/span> ROOT<\/span> \/<\/span> \"<\/span>oracle<\/span>\"<\/span> \/<\/span> \"<\/span>cases.json<\/span>\"<\/span> def<\/span> load_cases<\/span>()<\/span> -><\/span> list<\/span>[<\/span>dict<\/span>]:<\/span> return<\/span> json<\/span>.<\/span>loads<\/span>(<\/span>ORACLE<\/span>.<\/span>read_text<\/span>())[<\/span>\"<\/span>cases<\/span>\"<\/span>]<\/span> def<\/span> main<\/span>()<\/span> -><\/span> int<\/span>:<\/span> from<\/span> src.handler<\/span> import<\/span> handle<\/span> from<\/span> oracle.invariants<\/span> import<\/span> check<\/span> as<\/span> check_invariants<\/span> failures<\/span> =<\/span> []<\/span> for<\/span> case<\/span> in<\/span> load_cases<\/span>():<\/span> got<\/span> =<\/span> handle<\/span>(<\/span>case<\/span>[<\/span>\"<\/span>input<\/span>\"<\/span>])<\/span> try<\/span>:<\/span> check_invariants<\/span>(<\/span>got<\/span>)<\/span> except<\/span> AssertionError<\/span> as<\/span> exc<\/span>:<\/span> failures<\/span>.<\/span>append<\/span>(<\/span> {<\/span>\"<\/span>id<\/span>\"<\/span>:<\/span> case<\/span>[<\/span>\"<\/span>id<\/span>\"<\/span>],<\/span> \"<\/span>invariant<\/span>\"<\/span>:<\/span> str<\/span>(<\/span>exc<\/span>),<\/span> \"<\/span>got<\/span>\"<\/span>:<\/span> got<\/span>}<\/span> )<\/span> continue<\/span> if<\/span> got<\/span> !=<\/span> case<\/span>[<\/span>\"<\/span>expect<\/span>\"<\/span>]:<\/span> failures<\/span>.<\/span>append<\/span>(<\/span> {<\/span>\"<\/span>id<\/span>\"<\/span>:<\/span> case<\/span>[<\/span>\"<\/span>id<\/span>\"<\/span>],<\/span> \"<\/span>expected<\/span>\"<\/span>:<\/span> case<\/span>[<\/span>\"<\/span>expect<\/span>\"<\/span>],<\/span> \"<\/span>got<\/span>\"<\/span>:<\/span> got<\/span>}<\/span> )<\/span> if<\/span> failures<\/span>:<\/span> print<\/span>(<\/span>json<\/span>.<\/span>dumps<\/span>({<\/span>\"<\/span>pass<\/span>\"<\/span>:<\/span> False<\/span>,<\/span> \"<\/span>failures<\/span>\"<\/span>:<\/span> failures<\/span>},<\/span> indent<\/span>=<\/span>2<\/span>))<\/span> return<\/span> 1<\/span> print<\/span>(<\/span>json<\/span>.<\/span>dumps<\/span>({<\/span>\"<\/span>pass<\/span>\"<\/span>:<\/span> True<\/span>,<\/span> \"<\/span>count<\/span>\"<\/span>:<\/span> len<\/span>(<\/span>load_cases<\/span>())}))<\/span> return<\/span> 0<\/span> if<\/span> __name__<\/span> ==<\/span> \"<\/span>__main__<\/span>\"<\/span>:<\/span> raise<\/span> SystemExit<\/span>(<\/span>main<\/span>())<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
\n
# oracle\/invariants.py <\/span>def<\/span> check<\/span>(<\/span>result<\/span>:<\/span> dict<\/span>)<\/span> -><\/span> None<\/span>:<\/span> if<\/span> \"<\/span>remaining_cents<\/span>\"<\/span> in<\/span> result<\/span>:<\/span> assert<\/span> result<\/span>[<\/span>\"<\/span>remaining_cents<\/span>\"<\/span>]<\/span> >=<\/span> 0<\/span>,<\/span> \"<\/span>remaining_cents is negative<\/span>\"<\/span> if<\/span> result<\/span>.<\/span>get<\/span>(<\/span>\"<\/span>status<\/span>\"<\/span>)<\/span> ==<\/span> \"<\/span>ok<\/span>\"<\/span>:<\/span> assert<\/span> \"<\/span>refunded_cents<\/span>\"<\/span> in<\/span> result<\/span>,<\/span> \"<\/span>ok result missing refunded_cents<\/span>\"<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Run it before the agent starts work. Run it after every candidate patch. A red grader is information you wanted. <\/p>\n
\n
export <\/span>PYTHONPATH<\/span>=<\/span>.<\/span> python tools\/grade.py echo<\/span> $?<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
A rewritten oracle is contamination. Treat that diff as an incident. Do not chat the model through it.<\/p>\n
  <\/a> Mechanical freeze <\/h2>\n
Policy comments do not stop an agent loop. Git status does stop an agent loop. Enforce the freeze with a command. <\/p>\n
\n
# tools\/freeze_check.py <\/span>from<\/span> __future__<\/span> import<\/span> annotations<\/span> import<\/span> subprocess<\/span> import<\/span> sys<\/span> FROZEN<\/span> =<\/span> (<\/span> \"<\/span>oracle\/<\/span>\"<\/span>,<\/span> \"<\/span>tools\/grade.py<\/span>\"<\/span>,<\/span> \"<\/span>tools\/freeze_check.py<\/span>\"<\/span>,<\/span> )<\/span> def<\/span> changed_files<\/span>()<\/span> -><\/span> list<\/span>[<\/span>str<\/span>]:<\/span> staged<\/span> =<\/span> subprocess<\/span>.<\/span>check_output<\/span>(<\/span> [<\/span>\"<\/span>git<\/span>\"<\/span>,<\/span> \"<\/span>diff<\/span>\"<\/span>,<\/span> \"<\/span>--name-only<\/span>\"<\/span>,<\/span> \"<\/span>--cached<\/span>\"<\/span>],<\/span> text<\/span>=<\/span>True<\/span>,<\/span> )<\/span> unstaged<\/span> =<\/span> subprocess<\/span>.<\/span>check_output<\/span>(<\/span> [<\/span>\"<\/span>git<\/span>\"<\/span>,<\/span> \"<\/span>diff<\/span>\"<\/span>,<\/span> \"<\/span>--name-only<\/span>\"<\/span>],<\/span> text<\/span>=<\/span>True<\/span>,<\/span> )<\/span> names<\/span> =<\/span> set<\/span>()<\/span> for<\/span> block<\/span> in<\/span> (<\/span>staged<\/span>,<\/span> unstaged<\/span>):<\/span> names<\/span>.<\/span>update<\/span>(<\/span> line<\/span>.<\/span>strip<\/span>()<\/span> for<\/span> line<\/span> in<\/span> block<\/span>.<\/span>splitlines<\/span>()<\/span> if<\/span> line<\/span>.<\/span>strip<\/span>()<\/span> )<\/span> return<\/span> sorted<\/span>(<\/span>names<\/span>)<\/span> def<\/span> is_frozen<\/span>(<\/span>path<\/span>:<\/span> str<\/span>)<\/span> -><\/span> bool<\/span>:<\/span> return<\/span> any<\/span>(<\/span>path<\/span> ==<\/span> prefix<\/span> or<\/span> path<\/span>.<\/span>startswith<\/span>(<\/span>prefix<\/span>)<\/span> for<\/span> prefix<\/span> in<\/span> FROZEN<\/span>)<\/span> def<\/span> main<\/span>()<\/span> -><\/span> int<\/span>:<\/span> blocked<\/span> =<\/span> [<\/span>path<\/span> for<\/span> path<\/span> in<\/span> changed_files<\/span>()<\/span> if<\/span> is_frozen<\/span>(<\/span>path<\/span>)]<\/span> if<\/span> blocked<\/span>:<\/span> print<\/span>(<\/span>\"<\/span>frozen path edited:<\/span>\"<\/span>)<\/span> for<\/span> path<\/span> in<\/span> blocked<\/span>:<\/span> print<\/span>(<\/span>f<\/span>\"<\/span>- <\/span>{<\/span>path<\/span>}<\/span>\"<\/span>)<\/span> return<\/span> 2<\/span> print<\/span>(<\/span>\"<\/span>freeze check passed<\/span>\"<\/span>)<\/span> return<\/span> 0<\/span> if<\/span> __name__<\/span> ==<\/span> \"<\/span>__main__<\/span>\"<\/span>:<\/span> raise<\/span> SystemExit<\/span>(<\/span>main<\/span>())<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Wire one command for humans and CI: <\/p>\n
\n
python tools\/freeze_check.py &&<\/span> PYTHONPATH<\/span>=<\/span>.<\/span> python tools\/grade.py <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
If freeze check returns 2, reset frozen paths. Do not negotiate with the model. Restore the answer key immediately. <\/p>\n
\n
git checkout --<\/span> oracle tools\/grade.py tools\/freeze_check.py <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
  <\/a> Decision table <\/h2>\n
Paste this table into the pull request.<\/p>\n
\n\n\n\n\n\n\n\n\n\n\n\n
Observed state<\/th>\nAgent may edit src\/<\/code> <\/th>\nMerge?<\/th>\n<\/tr>\n<\/thead>\n
No oracle committed<\/td>\nNo<\/td>\nNo<\/td>\n<\/tr>\n
Freeze check dirty<\/td>\nNo<\/td>\nNo<\/td>\n<\/tr>\n
Grader red, oracle untouched<\/td>\nYes<\/td>\nNo<\/td>\n<\/tr>\n
Grader green, freeze clean<\/td>\nStop generating<\/td>\nReview only<\/td>\n<\/tr>\n
Patch includes oracle hunks<\/td>\nReject the patch<\/td>\nNo<\/td>\n<\/tr>\n
Snapshots regenerated with src<\/td>\nTreat as incident<\/td>\nNo<\/td>\n<\/tr>\n
Invariant assertion deleted<\/td>\nReject the patch<\/td>\nNo<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
If two rows apply, take the stricter row. Do not average conflicting rows. Strictness is the point of the table.<\/p>\n
  <\/a> Cheap generation still needs a local grade <\/h2>\n
You do not need a paid API here. You need retries, a branch, and freeze. Grade locally after every generated hunk.<\/p>\n
Disclosure: This article was prepared as part of MonkeyCode’s product outreach.<\/p>\n
MonkeyCode is an open-source coding assistant. It provides free model access and a free server option. Point it at src\/<\/code> only. Keep oracle\/<\/code> on your machine. Grade every candidate patch locally.<\/p>\n
A practical sequence looks like this: <\/p>\n
\n
git switch -c<\/span> agent\/refund-oracle PYTHONPATH<\/span>=<\/span>.<\/span> python tools\/grade.py ><\/span> \/tmp\/grade.json # give the assistant \/tmp\/grade.json<\/span> # do not grant write access to oracle\/<\/span> git apply --check<\/span> \/tmp\/src.patch git apply \/tmp\/src.patch python tools\/freeze_check.py ||<\/span> git checkout --<\/span> oracle tools PYTHONPATH<\/span>=<\/span>.<\/span> python tools\/grade.py <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Feed the failing JSON into the assistant. Do not grant oracle write permission. Hidden fixtures are a feature, not a problem.<\/p>\n
The free server is useful for patch generation. It is not useful as a grader host. Your laptop already knows the answer key.<\/p>\n
  <\/a> Reproducible test plan <\/h2>\n
Run this plan on a clean clone. No extra services are required today.<\/p>\n
    \n
  1. Commit the three oracle cases above.<\/li>\n
  2. Replace src\/handler.py<\/code> with a constant ok response.<\/li>\n
  3. Run the grader and confirm exit code 1.<\/li>\n
  4. Confirm the JSON lists all three case ids.<\/li>\n
  5. Restore reject paths only, then grade again.<\/li>\n
  6. Confirm refund-partial-usd<\/code> still fails loudly.<\/li>\n
  7. Restore remaining_cents math and expect exit 0.<\/li>\n
  8. Edit oracle remaining_cents to the wrong number.<\/li>\n
  9. Run freeze_check and confirm exit code 2.<\/li>\n
  10. Restore the oracle and confirm freeze_check exit 0.<\/li>\n<\/ol>\n
    If step 9 does not fail, your freeze is theater. Fix the path prefixes before inviting agents.<\/p>\n
    Add one pytest wrapper if your CI demands pytest. Keep it thin. The wrapper should call grade.py<\/code> and assert exit code 0. <\/p>\n
    \n
    # tests\/test_oracle_wiring.py <\/span>import<\/span> subprocess<\/span> import<\/span> sys<\/span> from<\/span> pathlib<\/span> import<\/span> Path<\/span> ROOT<\/span> =<\/span> Path<\/span>(<\/span>__file__<\/span>).<\/span>resolve<\/span>().<\/span>parents<\/span>[<\/span>1<\/span>]<\/span> def<\/span> test_grader_exits_zero<\/span>():<\/span> result<\/span> =<\/span> subprocess<\/span>.<\/span>run<\/span>(<\/span> [<\/span>sys<\/span>.<\/span>executable<\/span>,<\/span> \"<\/span>tools\/grade.py<\/span>\"<\/span>],<\/span> cwd<\/span>=<\/span>ROOT<\/span>,<\/span> env<\/span>=<\/span>{<\/span>**<\/span>dict<\/span>(<\/span>**<\/span>{<\/span>k<\/span>:<\/span> v<\/span> for<\/span> k<\/span>,<\/span> v<\/span> in<\/span> __import__<\/span>(<\/span>\"<\/span>os<\/span>\"<\/span>).<\/span>environ<\/span>.<\/span>items<\/span>()}),<\/span> \"<\/span>PYTHONPATH<\/span>\"<\/span>:<\/span> str<\/span>(<\/span>ROOT<\/span>)},<\/span> )<\/span> assert<\/span> result<\/span>.<\/span>returncode<\/span> ==<\/span> 0<\/span> <\/code><\/pre>\n
    \n<\/p><\/div>\n<\/p><\/div>\n
    That test proves wiring. It does not prove the agent is honest. Honesty comes from the frozen fixtures.<\/p>\n
      <\/a> Failure analysis <\/h2>\n
    When the grader stays red, read the JSON. Do not open a long chat thread.<\/p>\n