OpenAI’s COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden

COBOL is Astra’s lowest-scoring language, at 72.1% (OpenAI). Luna is the cheap option: OpenAI estimates $0.77 per sample against $3.48 for GPT-5.6 Sol, with much lower scores than Astra.

Price matters here because the two headline models sit far apart. GPT-6.1 Sol lists at $2 per million input tokens and $10 per million output. GPT-6 Astra lists at $10 and $50. For the migration eval, OpenAI says GPT-6.1 Sol ran at 61% of GPT-5.6 Sol’s estimated API cost (OpenAI). TokenPost notes that this figure covers the eval only, not the cost of moving a production system.

One half of this you can run

MultiLangBench is on GitHub under Apache-2.0. Its README says the model must implement SQLite 3.40.1 in the requested language, with SQLite’s documentation as the reference. The tests were converted from SQLite’s TCL suite, about 24k of roughly 51k. Each test is an initial database, a user query and a validation query.

A few rules in the README are worth knowing before you compare your numbers with OpenAI’s:

• Use only the 272 tests whose names start with 00, 01 or 02. The full suite is allowed for extra precision on high-scoring samples, as long as you say so.
• Strip every trace of SQLite from the environment first, and run with no internet access.
• Each test gets a 60-second timeout, and a solution in the wrong language scores 0.

The score is a pair, API cost and test pass rate, so a cheaper model with a lower score is not automatically worse for your budget (MultiLangBench).

The migration eval is the half you can’t rerun. Runtime Wire calls them “OpenAI-designed hidden tests,” and the GitHub repo covers only the SQLite build task. So the 86.6% is OpenAI’s measurement against OpenAI’s own hidden tests. OpenAI also says migration output still needs expert review and testing against business rules and system dependencies before it goes to production (OpenAI).

Build the parity check yourself

What you can reuse from OpenAI’s method is the pass rule: a migration counts only when the outputs match exactly. You can build that check on your own code before you trust any model with it.

Collect real inputs, run the old build and the migrated program on each one, and compare the bytes:

import subprocess import sys from pathlib import Path OLD = ["./legacy/payroll"] # the existing COBOL build NEW = ["python3", "payroll.py"] # the migrated program  def run(cmd, data): result = subprocess.run( cmd, input=data, capture_output=True, timeout=60 ) return result.returncode, result.stdout failed = 0 for case in sorted(Path("cases").glob("*.in")): data = case.read_bytes() if run(OLD, data) != run(NEW, data): failed += 1 print(f"DIFF {case.name}") print(f"{failed} failing cases") sys.exit(1 if failed else 0) 

It compares exit codes and stdout as raw bytes, so trailing spaces, line endings and padded numbers all count. If your batch job writes files instead of printing, compare those files the same way.

The cases that break migrations are the quiet ones. OpenAI’s example is COBOL’s fixed-point arithmetic, which truncates extra decimal digits unless rounding is requested (OpenAI). In Python, the decimal module lets you say which behaviour you mean:

from decimal import Decimal, ROUND_DOWN, ROUND_HALF_UP amount = Decimal("2") / Decimal("3") cents = Decimal("0.01") truncated = amount.quantize(cents, rounding=ROUND_DOWN) rounded = amount.quantize(cents, rounding=ROUND_HALF_UP) 

Two thirds truncates to 0.66 and rounds to 0.67. A migrated program that reaches for float and round() takes the rounding path, and every test that only checks “close enough” will pass it. Seed your cases/ folder with values like these, plus empty records and maximum field widths. TokenPost makes the same point: changes to rounding or to how missing records are handled can alter a result even when the replacement appears to work.

Running the migration as a task

A COBOL port is long agent work: read the program, write the new one, run the parity script, fix the diffs, run it again. Give it a branch of its own so the legacy tree stays untouched while the agent iterates.

I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex and other command-line agents side by side, each in its own terminal and its own worktree (README). ⌘N starts a task on its own hermes/<name> branch, with the task as the agent’s first prompt. That lets you hand the same migration ticket to two agents and run the parity script against both branches. I haven’t run OpenAI’s models on a COBOL migration myself.

Hermes IDE 2.3.4: Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with OpenAI, Anthropic or any agent vendor.

OpenAI is not alone here: TokenPost points to IBM’s COBOL-to-Java tool with comparison tests, and to Anthropic’s published approach to migrating COBOL with Claude Code. Whichever agent you pick, the parity script decides whether the port ships.

What to do this week

If you own COBOL, Fortran or Visual Basic code, three steps cost little:

1. Build the parity harness and a cases/ folder from real production inputs, before any model touches the code.
2. Run MultiLangBench on the 272-test subset for the language you care about, if you want a number of your own rather than OpenAI’s.
3. Try one small, well-tested program with GPT-6.1 Sol through the API, and count how many of your cases come back byte-identical.

Do you have legacy code you would let a model migrate today, if the parity harness came back clean? Tell me in the comments.

If this was useful

A migration like this is an agent loop: write, run the check, read the diff, try again, and stop when the harness is green. The AI Agents Pocket Guide covers the patterns behind that loop, from tool calling to guardrails that keep an agent from declaring victory early. If you want to run two agents on the same ticket, Hermes IDE is free and I build it.