AI migrated legacy COBOL programs to Java, bugs included

Who Should Read

Backend developers migrating legacy code with AI who need a way to check the result matches the original

Core Mechanics

• Run original and output side by side
  • The COBOL source and the generated Java are each instrumented with mocks and run off-mainframe on commodity hardware.
  • Deterministic parity checks compare outputs for the same inputs.
• Witness Search pushes through branches
  • An agent varies input mocks to get through program branches one by one.
  • Parity-preserving mutations follow.
• Locked Paragraphs explain the stalls
  • When input search can't go deeper, an analyzer identifies the condition blocking exploration.
  • That condition is the Locked Paragraph, and resolving it gets past the input-search plateau.
• Coverage improved in three case studies
  • Two open-source programs and one internal production-like COBOL program, 430 to 4,114 lines.
  • Nearly complete coverage on the open-source ones and 91.90% branch coverage on the internal one.
• Every accepted test matched

Community Reactions

• Opinion that COBOL is the whole mainframe environment
  • Commenters say 99.99% uptime and live CPU swapping aren't replaced by a language port.
  • Several asked how JCL, CICS, sort processors, and messaging map to Java.
• Criticism that the test scale is tiny
  • A commenter noted the largest case was 4kloc while production COBOL is hundreds of billions of lines.
  • The IRS was cited as having about 160 programs averaging 230kloc.
• Experience with an LLM inventing preprocessor behavior
  • A commenter fed real production code with a custom preprocessor to LLMs without an agent harness.
  • The models made up what it did, and Gemini 2.5 didn't even notice the preprocessor.
• Example of GOTO translation adding new bugs
  • A commenter showed that with COUNTER at 10, COBOL prints 11 while a while-loop translation prints nothing.
  • Similar tools mentioned include Mechanical Orchard's approach and the coboltwin comparison tool.

How to Apply

• Write a script comparing old and new code on the same inputs
  • Pick one function you're migrating today and build a harness that calls both implementations.
  • Feed 20 sample inputs and diff the outputs for exact equality.
• Replace external dependencies with mocks and run locally
  • Swap DB, messaging, and file calls for JSON-in/JSON-out mocks.
  • Make both implementations read the same mock data so runs are reproducible.
• Have an agent analyze uncovered branches
  • Extract branches never hit from your coverage report and give them to an LLM.
  • Ask what condition blocks each one and have it generate inputs that reach it.
• Set the pass criterion as output parity
  • Add a CI step that fails whenever the two implementations differ.
  • Accept LLM-generated tests only if they pass on the original first.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: blocked-branch analysis)

I'm migrating legacy code to a new language. I'll give you the original code and a coverage report. Do the following.

[Original code]
{{original code}}

[Branches never executed]
{{uncovered branches}}

[Instructions]
1. For each uncovered branch, explain in one line the condition blocking entry (the equivalent of a Locked Paragraph).
2. Write input mock values as JSON that would get through that condition.
3. State the expected output on the original. If you don't know, say so.

Next step: feed the inputs to both the original and the new implementation and diff the outputs.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://arxiv.org/abs/2607.28271
• https://www.mechanical-orchard.com/
• https://github.com/mikko-ahonen/coboltwin
• https://www.ibm.com/products/watsonx-code-assistant-z