{"id":6182,"date":"2026-10-20T04:31:17","date_gmt":"2026-10-20T07:31:17","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/10\/20\/openais-cobol-migration-score-jumped-from-5-4-to-86-6-the-tests-are-hidden\/"},"modified":"2026-10-20T04:31:17","modified_gmt":"2026-10-20T07:31:17","slug":"openais-cobol-migration-score-jumped-from-5-4-to-86-6-the-tests-are-hidden","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/10\/20\/openais-cobol-migration-score-jumped-from-5-4-to-86-6-the-tests-are-hidden\/","title":{"rendered":"OpenAI&#8217;s COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden"},"content":{"rendered":"<div>\n<div>\n<p>COBOL is Astra&#8217;s lowest-scoring language, at 72.1% (<a href=\"https:\/\/openai.com\/index\/low-resource-programming-languages\/\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI<\/a>). Luna is the cheap option: OpenAI estimates $0.77 per sample against $3.48 for GPT-5.6 Sol, with much lower scores than Astra.<\/p>\n<p>Price matters here because the two headline models sit far apart. <a href=\"https:\/\/developers.openai.com\/api\/docs\/models\/gpt-6.1-sol\" target=\"_blank\" rel=\"noopener noreferrer\">GPT-6.1 Sol<\/a> lists at $2 per million input tokens and $10 per million output. <a href=\"https:\/\/developers.openai.com\/api\/docs\/models\/gpt-6-astra\" target=\"_blank\" rel=\"noopener noreferrer\">GPT-6 Astra<\/a> lists at $10 and $50. For the migration eval, OpenAI says GPT-6.1 Sol ran at 61% of GPT-5.6 Sol&#8217;s estimated API cost (<a href=\"https:\/\/openai.com\/index\/low-resource-programming-languages\/\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI<\/a>). <a href=\"https:\/\/www.tokenpost.com\/news\/technology\/35357\" target=\"_blank\" rel=\"noopener noreferrer\">TokenPost<\/a> notes that this figure covers the eval only, not the cost of moving a production system.<\/p>\n<h2> <a name=\"one-half-of-this-you-can-run\" href=\"#one-half-of-this-you-can-run\"> <\/a> One half of this you can run <\/h2>\n<p><a href=\"https:\/\/github.com\/openai\/MultiLangBench\" target=\"_blank\" rel=\"noopener noreferrer\">MultiLangBench is on GitHub<\/a> under Apache-2.0. Its README says the model must implement SQLite 3.40.1 in the requested language, with SQLite&#8217;s documentation as the reference. The tests were converted from SQLite&#8217;s TCL suite, about 24k of roughly 51k. Each test is an initial database, a user query and a validation query.<\/p>\n<p>A few rules in the README are worth knowing before you compare your numbers with OpenAI&#8217;s:<\/p>\n<ul>\n<li>Use only the 272 tests whose names start with <code>00<\/code>, <code>01<\/code> or <code>02<\/code>. The full suite is allowed for extra precision on high-scoring samples, as long as you say so.<\/li>\n<li>Strip every trace of SQLite from the environment first, and run with no internet access.<\/li>\n<li>Each test gets a 60-second timeout, and a solution in the wrong language scores 0.<\/li>\n<\/ul>\n<p>The score is a pair, API cost and test pass rate, so a cheaper model with a lower score is not automatically worse for your budget (<a href=\"https:\/\/github.com\/openai\/MultiLangBench\" target=\"_blank\" rel=\"noopener noreferrer\">MultiLangBench<\/a>).<\/p>\n<p>The migration eval is the half you can&#8217;t rerun. <a href=\"https:\/\/runtimewire.com\/article\/openai-low-resource-language-coding-benchmarks\" target=\"_blank\" rel=\"noopener noreferrer\">Runtime Wire<\/a> calls them &#8220;OpenAI-designed hidden tests,&#8221; and the GitHub repo covers only the SQLite build task. So the 86.6% is OpenAI&#8217;s measurement against OpenAI&#8217;s own hidden tests. OpenAI also says migration output still needs expert review and testing against business rules and system dependencies before it goes to production (<a href=\"https:\/\/openai.com\/index\/low-resource-programming-languages\/\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI<\/a>).<\/p>\n<h2> <a name=\"build-the-parity-check-yourself\" href=\"#build-the-parity-check-yourself\"> <\/a> Build the parity check yourself <\/h2>\n<p>What you can reuse from OpenAI&#8217;s method is the pass rule: a migration counts only when the outputs match exactly. You can build that check on your own code before you trust any model with it.<\/p>\n<p>Collect real inputs, run the old build and the migrated program on each one, and compare the bytes: <\/p>\n<div>\n<pre><code><span>import<\/span> <span>subprocess<\/span> <span>import<\/span> <span>sys<\/span> <span>from<\/span> <span>pathlib<\/span> <span>import<\/span> <span>Path<\/span> <span>OLD<\/span> <span>=<\/span> <span>[<\/span><span>\"<\/span><span>.\/legacy\/payroll<\/span><span>\"<\/span><span>]<\/span> <span># the existing COBOL build <\/span><span>NEW<\/span> <span>=<\/span> <span>[<\/span><span>\"<\/span><span>python3<\/span><span>\"<\/span><span>,<\/span> <span>\"<\/span><span>payroll.py<\/span><span>\"<\/span><span>]<\/span> <span># the migrated program <\/span> <span>def<\/span> <span>run<\/span><span>(<\/span><span>cmd<\/span><span>,<\/span> <span>data<\/span><span>):<\/span> <span>result<\/span> <span>=<\/span> <span>subprocess<\/span><span>.<\/span><span>run<\/span><span>(<\/span> <span>cmd<\/span><span>,<\/span> <span>input<\/span><span>=<\/span><span>data<\/span><span>,<\/span> <span>capture_output<\/span><span>=<\/span><span>True<\/span><span>,<\/span> <span>timeout<\/span><span>=<\/span><span>60<\/span> <span>)<\/span> <span>return<\/span> <span>result<\/span><span>.<\/span><span>returncode<\/span><span>,<\/span> <span>result<\/span><span>.<\/span><span>stdout<\/span> <span>failed<\/span> <span>=<\/span> <span>0<\/span> <span>for<\/span> <span>case<\/span> <span>in<\/span> <span>sorted<\/span><span>(<\/span><span>Path<\/span><span>(<\/span><span>\"<\/span><span>cases<\/span><span>\"<\/span><span>).<\/span><span>glob<\/span><span>(<\/span><span>\"<\/span><span>*.in<\/span><span>\"<\/span><span>)):<\/span> <span>data<\/span> <span>=<\/span> <span>case<\/span><span>.<\/span><span>read_bytes<\/span><span>()<\/span> <span>if<\/span> <span>run<\/span><span>(<\/span><span>OLD<\/span><span>,<\/span> <span>data<\/span><span>)<\/span> <span>!=<\/span> <span>run<\/span><span>(<\/span><span>NEW<\/span><span>,<\/span> <span>data<\/span><span>):<\/span> <span>failed<\/span> <span>+=<\/span> <span>1<\/span> <span>print<\/span><span>(<\/span><span>f<\/span><span>\"<\/span><span>DIFF <\/span><span>{<\/span><span>case<\/span><span>.<\/span><span>name<\/span><span>}<\/span><span>\"<\/span><span>)<\/span> <span>print<\/span><span>(<\/span><span>f<\/span><span>\"<\/span><span>{<\/span><span>failed<\/span><span>}<\/span><span> failing cases<\/span><span>\"<\/span><span>)<\/span> <span>sys<\/span><span>.<\/span><span>exit<\/span><span>(<\/span><span>1<\/span> <span>if<\/span> <span>failed<\/span> <span>else<\/span> <span>0<\/span><span>)<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>It compares exit codes and stdout as raw bytes, so trailing spaces, line endings and padded numbers all count. If your batch job writes files instead of printing, compare those files the same way.<\/p>\n<p>The cases that break migrations are the quiet ones. OpenAI&#8217;s example is COBOL&#8217;s fixed-point arithmetic, which truncates extra decimal digits unless rounding is requested (<a href=\"https:\/\/openai.com\/index\/low-resource-programming-languages\/\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI<\/a>). In Python, <a href=\"https:\/\/docs.python.org\/3\/library\/decimal.html\" target=\"_blank\" rel=\"noopener noreferrer\">the <code>decimal<\/code> module<\/a> lets you say which behaviour you mean: <\/p>\n<div>\n<pre><code><span>from<\/span> <span>decimal<\/span> <span>import<\/span> <span>Decimal<\/span><span>,<\/span> <span>ROUND_DOWN<\/span><span>,<\/span> <span>ROUND_HALF_UP<\/span> <span>amount<\/span> <span>=<\/span> <span>Decimal<\/span><span>(<\/span><span>\"<\/span><span>2<\/span><span>\"<\/span><span>)<\/span> <span>\/<\/span> <span>Decimal<\/span><span>(<\/span><span>\"<\/span><span>3<\/span><span>\"<\/span><span>)<\/span> <span>cents<\/span> <span>=<\/span> <span>Decimal<\/span><span>(<\/span><span>\"<\/span><span>0.01<\/span><span>\"<\/span><span>)<\/span> <span>truncated<\/span> <span>=<\/span> <span>amount<\/span><span>.<\/span><span>quantize<\/span><span>(<\/span><span>cents<\/span><span>,<\/span> <span>rounding<\/span><span>=<\/span><span>ROUND_DOWN<\/span><span>)<\/span> <span>rounded<\/span> <span>=<\/span> <span>amount<\/span><span>.<\/span><span>quantize<\/span><span>(<\/span><span>cents<\/span><span>,<\/span> <span>rounding<\/span><span>=<\/span><span>ROUND_HALF_UP<\/span><span>)<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<p>Two thirds truncates to 0.66 and rounds to 0.67. A migrated program that reaches for <code>float<\/code> and <code>round()<\/code> takes the rounding path, and every test that only checks &#8220;close enough&#8221; will pass it. Seed your <code>cases\/<\/code> folder with values like these, plus empty records and maximum field widths. <a href=\"https:\/\/www.tokenpost.com\/news\/technology\/35357\" target=\"_blank\" rel=\"noopener noreferrer\">TokenPost<\/a> makes the same point: changes to rounding or to how missing records are handled can alter a result even when the replacement appears to work.<\/p>\n<h2> <a name=\"running-the-migration-as-a-task\" href=\"#running-the-migration-as-a-task\"> <\/a> Running the migration as a task <\/h2>\n<p>A COBOL port is long agent work: read the program, write the new one, run the parity script, fix the diffs, run it again. Give it a branch of its own so the legacy tree stays untouched while the agent iterates.<\/p>\n<p>I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex and other command-line agents side by side, each in its own terminal and its own worktree (<a href=\"https:\/\/github.com\/hermes-hq\/hermes-ide\" target=\"_blank\" rel=\"noopener noreferrer\">README<\/a>). \u2318N starts a task on its own <code>hermes\/&lt;name&gt;<\/code> branch, with the task as the agent&#8217;s first prompt. That lets you hand the same migration ticket to two agents and run the parity script against both branches. I haven&#8217;t run OpenAI&#8217;s models on a COBOL migration myself.<\/p>\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvubo4xjvjq8dhkk462u3.png\"><\/a><br \/> <em>Hermes IDE 2.3.4: Codex and Claude Code side by side, each in its own hermes\/ worktree, with status, context use, cost and branch in the sidebar.<\/em><\/p>\n<p>Hermes is free to <a href=\"https:\/\/hermes-ide.com\/download\" target=\"_blank\" rel=\"noopener noreferrer\">download<\/a>, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with OpenAI, Anthropic or any agent vendor.<\/p>\n<p>OpenAI is not alone here: <a href=\"https:\/\/www.tokenpost.com\/news\/technology\/35357\" target=\"_blank\" rel=\"noopener noreferrer\">TokenPost<\/a> points to IBM&#8217;s COBOL-to-Java tool with comparison tests, and to Anthropic&#8217;s published approach to migrating COBOL with Claude Code. Whichever agent you pick, the parity script decides whether the port ships.<\/p>\n<h2> <a name=\"what-to-do-this-week\" href=\"#what-to-do-this-week\"> <\/a> What to do this week <\/h2>\n<p>If you own COBOL, Fortran or Visual Basic code, three steps cost little:<\/p>\n<ol>\n<li>Build the parity harness and a <code>cases\/<\/code> folder from real production inputs, before any model touches the code.<\/li>\n<li>Run <a href=\"https:\/\/github.com\/openai\/MultiLangBench\" target=\"_blank\" rel=\"noopener noreferrer\">MultiLangBench<\/a> on the 272-test subset for the language you care about, if you want a number of your own rather than OpenAI&#8217;s.<\/li>\n<li>Try one small, well-tested program with <a href=\"https:\/\/developers.openai.com\/api\/docs\/models\/gpt-6.1-sol\" target=\"_blank\" rel=\"noopener noreferrer\">GPT-6.1 Sol through the API<\/a>, and count how many of your cases come back byte-identical.<\/li>\n<\/ol>\n<p>Do you have legacy code you would let a model migrate today, if the parity harness came back clean? Tell me in the comments.<\/p>\n<hr>\n<h2> <a name=\"if-this-was-useful\" href=\"#if-this-was-useful\"> <\/a> If this was useful <\/h2>\n<p>A migration like this is an agent loop: write, run the check, read the diff, try again, and stop when the harness is green. The <a href=\"https:\/\/www.amazon.com\/dp\/B0GYJZ2XJD\" target=\"_blank\" rel=\"noopener noreferrer\">AI Agents Pocket Guide<\/a> covers the patterns behind that loop, from tool calling to guardrails that keep an agent from declaring victory early. If you want to run two agents on the same ticket, <a href=\"https:\/\/hermes-ide.com\/download\" target=\"_blank\" rel=\"noopener noreferrer\">Hermes IDE<\/a> is free and I build it.<\/p>\n<p><a href=\"https:\/\/www.amazon.com\/dp\/B0GYJZ2XJD\" target=\"_blank\" rel=\"noopener noreferrer\"><\/a><\/p>\n<\/p><\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>Fuente: <a href=\"https:\/\/dev.to\/gabrielanhaia\/openais-cobol-migration-score-jumped-from-54-to-866-the-tests-are-hidden-29go\">Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>COBOL is Astra&#8217;s lowest-scoring language, at 72.1% (OpenAI). Luna is the cheap option: OpenAI estimates $0.77 per sample against $3.48 for GPT-5.6 Sol, with much lower scores than Astra. Price matters here because the two headline models sit far apart. GPT-6.1 Sol lists at $2 per million input tokens and $10 per million output. GPT-6 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":6181,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"webixso_pending_account_ids":""},"categories":[41],"tags":[],"class_list":["post-6182","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/6182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=6182"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/6182\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/6181"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=6182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=6182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=6182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}