The Study That Proved AI Slows Coders Down Just Reversed Itself

For over a year, one number anchored every argument about whether AI coding tools actually work: METR's 2025 study found that experienced developers took 19% longer to finish tasks when they were allowed to use AI. In February, METR reran it. The new number is a speedup. METR itself says you shouldn't trust it either.

Accurate as of September 2026.

Here's what changed between the two runs.

The headline flip reads like vindication for AI vendors, but the confidence interval undercuts it: the 2026 result spans a 38% speedup to a 9% slowdown. That range is too wide to plan around.

More telling is why METR says the number is unreliable. In its own writeup, the team lists four problems that broke the experiment:

1. Recruitment collapsed. Developers increasingly refused to work without AI at all, even for $50 an hour, which is well above market rate for this kind of task.
2. Task selection bias. Between 30% and 50% of participants withheld tasks they expected AI to accelerate, so the "no AI" condition got skewed toward harder work.
3. Quality and documentation diverged between the AI-allowed and AI-disallowed conditions, making the two arms less comparable than they looked.
4. Time tracking broke down once developers started running AI agents concurrently in the background while doing something else.

METR's own conclusion: "Our data is only very weak evidence for the size of this increase." They're redesigning the study around observational data and developer-level randomization instead, because the clean RCT design no longer works.

That's the actual story. A randomized controlled trial needs a credible control group willing to do the work the old way. In mid-2025, that group still existed. By late 2025, professional developers would rather turn down $50 an hour than code without AI assistance, a shift TechCrunch covered directly. You can't randomize away a habit that's become the default way people do their job. The control condition is now the artificial one.

This should change how you read every productivity claim from here forward, including your own. If you're trying to measure whether your team's AI coding setup is paying off, don't run an internal A/B test - you'll hit the exact selection bias METR did, because your best engineers will find a reason to use the tool on the tasks that make it look good. Track outcomes you already collect instead: cycle time from PR open to merge, revert rate, review turnaround. Google's 2025 DORA report found AI acts as an amplifier of whatever engineering discipline already exists on a team, not a fixed multiplier you can point at. That framing has held up better than either of METR's numbers.

Stop asking how much faster AI makes your team. Track what your review and deploy pipeline does with the extra code it's already producing - cycle time and revert rate will still mean something in six months, when this quarter's productivity number has been revised again.