Why would you deliberately preserve bugs when migrating legacy code?

Because after thirty years, the bugs are part of the interface. Downstream systems, batch jobs, reports and manual processes have all been built against the mainframe's actual behaviour, including the places where that behaviour is wrong. A rounding error that consistently drops a fraction of a cent has been silently reconciled somewhere else for decades. A date routine that mishandles a specific edge case has a compensating workaround in another department's spreadsheet. Fixing those during migration does not produce a correct system; it produces a system whose outputs no longer match what every consumer expects, and the resulting failures surface far from the change that caused them. The safe sequence is to migrate to exact behavioural equivalence first, then fix bugs deliberately, one at a time, with the consumers identified.

What is the Locksmith Loop?

It is the agentic test-synthesis method proposed in arXiv paper 2607.28271, 'Agentic Method for Deterministic Validation of Legacy Code Migration'. Rather than trying to prove the migrated Java is correct in the abstract, it executes both the original COBOL and the target Java on commodity hardware and compares their behaviour, iteratively synthesising test inputs to expand coverage. The validation target is behavioural equivalence with the legacy implementation, explicitly including that implementation's bugs, rather than conformance to a specification that in most cases no longer exists.

Why is test data the hard part of a legacy migration?

Because legacy systems typically have no test suite, no specification, and production data that cannot legally or safely be copied into a test environment. The behaviour that needs preserving is encoded only in the source itself and in decades of accumulated production runs. That leaves you unable to answer the basic question — does the new system do what the old one did? — because you have nothing to run it against. Synthesising inputs that exercise the corner cases is therefore not a convenience; it is the blocking problem, and it is what the Locksmith Loop is built to attack.

Can AI already translate COBOL to Java accurately?

Generation is largely handled; verification is not. One study of AI-driven COBOL-to-Java modernisation reports 93% accuracy against 75% for manual effort and 82% for rule-based tools, alongside a 35% drop in complexity and a 33% drop in coupling. A separate study found that large models successfully captured the original business logic and produced clean, genuinely object-oriented Java — but stated plainly that complete functional correctness could not be verified in practice, and that manual input-data preparation and extensive prompt engineering were required. Read together, the literature says the models write plausible Java and nobody can prove it behaves like the COBOL.

What does 93% accuracy actually mean for a migration project?

For a mainframe workload, it means the project is not finished. A financial batch system running millions of transactions does not have a tolerance for a 7% behavioural divergence; a single mismatched rounding rule can produce a reconciliation break that takes weeks to trace. The figure is useful as a comparison against manual and rule-based baselines, and it is a real improvement over both. It is not a deployment threshold. The gap between 93% and the required standard is precisely the gap that verification tooling has to close, which is why an equivalence-testing method matters more than another few points of generation accuracy.

How does this change the way migration projects should be run?

It moves the centre of gravity from translation to differential testing. Under the old framing, the project plan is 'convert the code, then test it', and testing is a phase at the end that always overruns. Under the new framing, the harness that runs both systems against synthesised inputs and compares outputs is the first thing you build, and translation becomes the cheap, iterable part. That reordering also changes what you buy: the scarce capability is no longer a code translator, it is an equivalence oracle.

What are the limits of this approach?

It requires both the source and the target to be executable, which rules out systems whose original environment can no longer be stood up — a real constraint with older mainframe estates. Equivalence is only ever established over the inputs actually tested, so confidence is bounded by the coverage the synthesis achieves, and there is no guarantee it finds every corner case. Systems with external side effects, non-deterministic behaviour, or dependence on specific hardware arithmetic complicate the comparison. And it validates behaviour, not architecture: a system can be perfectly equivalent and still be a poor design, which is a separate problem the method does not address.

The Right Way to Migrate COBOL Is to Keep the Bugs

A paper on the Hacker News front page this week carried the headline “AI migrated legacy COBOL programs to Java, bugs included.” It drew 29 points and 24 comments — near the bottom of the page, well behind a 2.4-trillion-parameter model announcement with no benchmarks and a discussion about kitchen knives.

That is a mispricing. The paper, arXiv:2607.28271, quietly reframes what an AI-assisted legacy migration is for, and it targets a market measured in billions of dollars of annual services spend.

The reframing is contained in the phrase everyone read as a failure. “Bugs included” is not the paper’s disclaimer. It is the paper’s goal.

Executive summary

The paper proposes the “Locksmith Loop”: an agentic test-synthesis method that runs both the COBOL source and the migrated Java on commodity hardware and iteratively compares behaviour.
The validation target is bug-for-bug equivalence with the legacy system, not correctness against a specification.
This is the right target, because after decades in production the legacy system’s bugs have been absorbed into the behaviour that downstream consumers depend on.
Adjacent work reports 93% accuracy for AI-driven COBOL-to-Java conversion against 75% manual and 82% rule-based — improvements that are real but nowhere near a deployment threshold.
A separate study found models produce clean, logically correct object-oriented Java but that complete functional correctness could not be verified in practice.
The bottleneck in legacy modernisation has moved from generation to verification. This paper attacks the bottleneck.

Why “correct” is the wrong target

Start with the thing that makes mainframe migration different from ordinary refactoring: there is usually no specification.

The system was written decades ago. The people who wrote it have retired. The requirements documents, if they ever existed, describe a business that has since reorganised twice. What the system does is defined by exactly one artefact — the source code — and by the accumulated record of what it has produced in production.

Now consider what that means for a bug. Suppose a currency routine rounds in a way that consistently loses a fraction of a cent on a particular class of transaction. In a system that has been running since the 1990s, that behaviour is not sitting in isolation waiting to be discovered. Somewhere downstream, a reconciliation process has been tuned to expect the discrepancy. A monthly report has a line item that only balances because of it. Possibly someone in finance has a spreadsheet with a correction column and no memory of why it is there.

Fix that bug during migration and you have not improved the system. You have changed its output, and every consumer calibrated against the old output now breaks — but they break subtly, downstream, weeks later, in processes nobody connected to the migration. The failure surfaces as far as possible from its cause, which is the most expensive shape a failure can have.

This generalises. Date-handling quirks, truncation behaviour, sort stability, the precise ordering of records in a batch output, the exact format of a padded field: all of it has been depended upon by something. In a system of that age, observable behaviour is the interface, and the distinction between intended behaviour and accidental behaviour has stopped being meaningful to anyone downstream.

So the correct migration target is not “Java that is right.” It is “Java that is identical, including where identical means wrong.” Bugs get fixed afterwards, deliberately, one at a time, with the affected consumers identified first. That sequencing is the difference between a migration that lands and one that gets rolled back.

What the Locksmith Loop does

The paper’s contribution is a way to establish that equivalence when you have almost nothing to test with.

The blocking problem in these projects is test data. There is no existing test suite. Production data frequently cannot be moved into a test environment for regulatory reasons. And even with data in hand, the corner cases — the inputs that expose the behavioural quirks that matter most — are by definition rare, so a sample of ordinary production traffic will not exercise them.

The Locksmith Loop attacks this by making test synthesis the iterative core of the process. It executes both the COBOL source and the Java target on commodity hardware, compares their behaviour, and uses the comparison to drive the synthesis of further inputs. Where the two implementations diverge, that is a defect in the migration. Where they agree, coverage extends.

Two design decisions are worth drawing out.

The first is commodity hardware. Requiring mainframe time for every validation cycle would make the loop economically unusable — mainframe capacity is expensive and rationed, and a method that needs it once per iteration cannot iterate. Getting the COBOL side running on ordinary machines is what turns validation from a phase into a loop.

The second is that the oracle is the legacy system itself. There is no specification to check against, so the paper stops pretending there is one and uses the only ground truth that actually exists. This is why the method can honestly report that the Java reproduces the legacy behaviour including its bugs — it is not detecting a failure, it is confirming success.

The state of the surrounding literature

Put the paper next to the adjacent work and a clear picture emerges.

One study of AI-driven COBOL-to-Java modernisation (arXiv:2504.11335) reports 93% accuracy, against 75% for manual effort and 82% for rule-based tools, with complexity down 35% and coupling down 33%. Those are meaningful gains over both baselines.

A separate study reached a more pointed conclusion: large models can successfully capture the original business logic and convert COBOL applications into logically correct, clean, genuinely object-oriented Java — but the complete functional correctness of the generated Java could not be verified in practice, and obtaining even those results required manual preparation of input data and extensive prompt engineering.

Read those two findings together and the situation is unambiguous. The models write good-looking Java. Nobody can prove it behaves like the COBOL.

And for this workload, 93% is not a passing grade. A batch system processing millions of financial transactions has no tolerance for a 7% behavioural divergence; one mismatched rounding rule produces a reconciliation break that consumes weeks of investigation. The number is a useful comparison against baselines and a poor deployment criterion. The distance between 93% and what a bank would actually sign off on is exactly the distance that verification tooling has to cover.

The bottleneck moved, and the tooling has not caught up

This is the shift worth internalising, and it extends well past COBOL.

For most of the last few years, the interesting question in AI-assisted code work was whether the model could produce the code. That question is now substantially answered across a wide range of tasks. The interesting question has become whether anyone can establish that the produced code is right — and on that, the tooling is thin.

In everyday development this shows up as review burden: teams generating code faster than they can meaningfully verify it. In legacy migration it shows up in its most acute form, because the volume is enormous, the stakes are financial and regulatory, and there is no specification to check against. The same underlying gap, with the safety margin removed.

Which is why the practical structure of these projects should invert. The instinct is to plan them as “convert the code, then test it,” with testing as a phase at the end. It always overruns, because the team discovers during that phase that they have no way to answer the central question. The better structure is to build the differential-testing harness first — the thing that runs both systems against synthesised inputs and diffs the results — and treat translation as the cheap, repeatable step it has become. When verification is a loop rather than a phase, a failed translation costs an iteration instead of a milestone.

That also changes what is worth buying, or building. The scarce capability in this market is no longer a COBOL-to-Java translator; several of those exist and the models are increasingly adequate at the job unaided. The scarce capability is an equivalence oracle.

Where it does not reach

The method has real limits, and they are worth stating.

It requires both sides to be executable. Systems whose original runtime environment can no longer be reconstructed — not rare in older estates — are outside its reach. Equivalence is established only over the inputs actually tested, so confidence is bounded by whatever coverage the synthesis achieves, and no synthesis strategy guarantees it finds every corner case. External side effects, non-determinism, and dependence on specific hardware arithmetic all complicate the comparison and may defeat it outright.

And it validates behaviour, not design. A migrated system can be perfectly equivalent to its predecessor and still be a bad system — the same tangled control flow, now in Java. Behavioural equivalence is the right first target precisely because it is narrow, but nobody should mistake it for modernisation. It is the safe landing that makes modernisation possible afterwards.

Why this deserved more than 29 points

The visible AI news cycle rewards capability announcements. A larger model, a longer autonomous run, a new frontier score. Verification work does not photograph well, and this paper’s headline made it sound like a bug report.

But the constraint on deploying AI-generated code in consequential systems is not model capability, and has not been for a while. It is the absence of a way to establish that the output is safe to run. This paper takes one high-value instance of that problem and supplies the missing primitive — and does so with an insight most practitioners get wrong on first contact, which is that in a thirty-year-old system, faithfulness beats correctness.

That is worth more attention than a parameter count.