The harder question behind AI’s productivity gains

METR’s early-2025 experiment found that experienced open-source developers completed their assigned tasks 19 percent more slowly with AI access. Its February 2026 follow-up is a warning against carrying that finding forward as a permanent verdict. The researchers said the new study could not reliably establish the tools’ current productivity effect.[1][2]

Some developers declined to participate because they did not want to work without AI. Reduced compensation introduced another possible selection effect, while developers running several agents made time measurement harder. The remaining participants were no longer a clean window into everyone using the tools.[2]

The raw follow-up estimates pointed toward faster completion, but their confidence intervals included no improvement. METR said conversations with participants suggested greater benefits than before, while stressing that the data offered only weak evidence about the size of that change.[2]

Measure the finished work

The distinction matters for anyone evaluating a coding assistant. Generating code, finishing a task and maintaining the resulting software are separate outcomes. A benchmark can measure one precisely while leaving the others almost untouched.

A practical comparison needs to include review, correction and integration time, alongside the work that the developer would otherwise have done. It should also ask whether the tool changes the task itself. Work that becomes cheap enough to attempt may create value without appearing as a speedup on an existing task list.

This is a research retrospective, not a claim about the performance of today’s particular models. Its enduring contribution is methodological: when adoption changes who will enter an experiment, even a careful study can become less representative of the population it hopes to describe.