METR 2025 RCT: AI Tools Slowed Experienced Open-Source Devs by 19% — Analysis Saved from a study session on 2026-07-08. Source: arXiv 2507.09089v2 — "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (Becker, Rush, Barnes & Rein, METR, July 2025). PDF:  Clippings/2507.09089v2.pdf What the study is METR ran a  randomized controlled trial  — the first real RCT of AI coding tools on  real  work — between February and June 2025: 16 experienced open-source developers , working on repos they'd contributed to for ~5 years (avg. 1,500 commits). These are big, mature projects: ~23k stars, ~1.1M lines of code, strict review standards. They brought  246 real issues  from their own repos — bugs, features, ~2 hours of work each. Each issue was randomized by coin flip : AI-allowed (mostly Cursor Pro + Claude 3.5/3.7 Sonnet) or AI-disallowed. Devs recorded their screens and reported time spent. The clever part of the design is  when  things get measured. Before randomization, developers forecast how long each issue would take both with and without AI. That does two jobs: it gives a difficulty measure that's independent of the coin flip, and it captures expectations. The outcome — time to complete the issue — is  fixed before treatment assignment , so AI can't game the metric. Compare that to earlier studies which measured lines of code or number of PRs — things AI can inflate without any real productivity gain (more verbose code, work split into more PRs). The headline result Three numbers, and the gap between them is the whole story: Who Expected effect of AI Economics experts 39% faster ML experts 38% faster The developers themselves (before) 24% faster The developers themselves ( after doing the work ) 20% faster What actually happened 19% slower Everyone got the  sign  wrong. And the most unsettling row is the fourth one: after spending hours using the AI on real tasks, developers still believed it had sped them up by 20% — while it had actually slowed them down. Interestingly, the devs' forecasts of  task duration  were well-calibrated (correlation ~0.6 with actual times) — they know how long work takes; what's inverted is specifically their belief about AI's contribution. Where the time went They hand-labeled 143 hours of screen recordings at ~10-second resolution. With AI allowed, developers spent  less  time actively coding and searching, and instead spent time  prompting the AI, waiting on generations, reviewing AI output  — and notably more time  idle . The AI didn't remove work; it swapped coding time for AI-babysitting time, and the swap was net negative in this setting. Why? The factor analysis They examined 21 possible explanations, grouped into four types (direct productivity loss / experimental artifact / raises human performance / limits AI performance). Five factors had real evidence  for  contributing to slowdown: Over-optimism  — devs believed AI was helping, so they kept using it past the point of usefulness. High familiarity  — these devs are experts  in these exact repos ; it's hard for AI to beat someone with all the context already in their head. Slowdown was worse on tasks devs knew best. Large, complex repos  — AI performs worse in million-line codebases than in greenfield projects. Low AI reliability  — devs accepted  <44%  of generations; 75% read every line of AI output; 56% often made major cleanups. ~9% of AI-allowed time went just to reviewing/cleaning AI output. Missing tacit context  — "AI acts like a new contributor": it doesn't know the undocumented constraints, the weird backwards-compat case, which location is the  right  place for an edit. Six factors had evidence  against  (cheating, dropout, non-frontier models, unfamiliar IDE, etc.), and ten were unclear. The authors are careful: they can't fully rule out experimental artifacts, but the slowdown was robust across many alternative analyses. The caveat they insist on This does  not  say "AI doesn't speed up developers." It says: for  experts  working in  codebases they know deeply  with  high quality bars , early-2025 AI slowed them down. The same paper explicitly says results are consistent with big speedups on greenfield projects or unfamiliar code — and their appendix quotes back this up: devs found AI  most  helpful precisely on tasks they'd never done before ("first time with Git hooks, AI saved me 3 hours"). Open discussion threads The perception gap  — developers finished the study still believing they'd been sped up 20%. How can a tool slow you down while feeling helpful? (Candidate hypothesis: "trading speed for ease" — effort vs. time.) Self-mapping  — for a heavy AI-tool user: which side of the familiarity line are you usually on, and do you recognize the review-and-cleanup tax in your own sessions? Design critique — 16 developers, issues capped at ~2 hours: what does that exclude?