Is AI making developers faster? What the data says

In early 2025, a research group called METR ran a careful experiment. Sixteen experienced open-source developers worked through 246 real tasks in projects they knew well. Each task was randomly assigned: AI tools allowed, or not. Before starting, the developers expected AI to make them 24% faster. When the study ended, they believed it had made them 20% faster.

The measured result: tasks done with AI took 19% longer.

That gap between how AI feels and what a stopwatch shows is the most useful thing to understand before you believe any team’s speed claims, including ours. This post goes through what the main studies found, why the feeling and the measurement drift apart, and what that means for how a team should use AI.

What METR measured in 2025

METR’s design is what makes the result hard to dismiss. The developers had contributed to their repositories for years. The projects averaged more than 22,000 GitHub stars and over a million lines of code. Tasks were real issues, bug fixes, features and refactors, averaging about two hours each. Developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, which were frontier models at the time, and they recorded their screens.

METR is careful about what the study does not show. It doesn’t claim AI slows down most developers, or that these developers represent most software work. It doesn’t rule out better ways of using the tools. It covers one setting: experts in large codebases they already know deeply, with early-2025 tools.

What changed by 2026

In February 2026, METR published an update. It had run the experiment again with late-2025 tools: 57 developers, 143 repositories, more than 800 tasks.

The estimates moved toward a speedup. For developers returning from the first study, AI tasks took an estimated 18% less time. For new developers, 4% less. Both confidence intervals still include zero, so neither result is statistically clear.

The more interesting finding is why METR calls the new data unreliable. Developers increasingly refused to take part because they didn’t want to work without AI, and many skipped tasks they didn’t want to do by hand. When surveyed, 30% to 50% of developers said they had done this. METR concludes developers are probably faster with AI in early 2026 than in early 2025, but its data is only very weak evidence of how much. The true speedup could be much higher for the people and tasks that dropped out.

So the honest summary is uncertain in both directions. The best controlled study we have found AI slowed experts down in early 2025, and can no longer run a clean comparison because developers won’t give the tools up.

What developers report

The Stack Overflow Developer Survey 2025 asked tens of thousands of developers about AI:

Finding	Share
Using or planning to use AI tools	84%
Professional developers using AI daily	51%
Distrust the accuracy of AI output	46%
Trust it	33%
Top frustration: answers that are “almost right, but not quite”	66%
Debugging AI-generated code takes more time	45%

Usage keeps rising while trust falls. The two biggest complaints are both about the same thing: code that looks finished and isn’t.

What happens to delivery

Google’s DORA research program surveyed nearly 5,000 technology professionals for its 2025 report on AI-assisted software development. Nearly 90% use AI at work. For the first time, DORA found AI adoption linked to higher delivery throughput. It also found software delivery instability keeps rising.

DORA’s framing is that AI is an amplifier. It magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones. A team with good tests, small changes and fast review ships more with AI. A team without those ships more problems.

What happens to security

Veracode tested more than 100 large language models on 80 coding tasks in Java, Python, C# and JavaScript, and published the results in July 2025. AI-generated code introduced security vulnerabilities in 45% of cases. Java failed more than 70% of the time. The models failed to defend against cross-site scripting in 86% of cases and log injection in 88%.

One finding should worry anyone waiting for better models to fix this. Veracode reports that security performance has stayed flat over time, and larger models did not do significantly better than smaller ones.

Why it feels faster than it is

A simple model shows where the gap comes from. It’s an illustration, not data from any of the studies.

Take a two-hour task, about the average size in METR’s study. Without AI, most of the visible effort is typing code and reading documentation. Those are the parts that feel like work.

With AI, the typing mostly disappears, which is very noticeable. New time appears in places that don’t feel like work:

• Writing and rewriting prompts
• Waiting for the model
• Reading generated code closely enough to trust it
• Fixing the “almost right” parts, which 66% of developers named as their top frustration

Each of those feels small, and none feels like the slow, effortful part of programming. Add them up and the task can take longer while feeling shorter. That fits METR’s 2025 result: developers who were measurably slower still believed they had been 20% faster.

The model also explains when AI helps most. Work with lots of typing and little need for deep review (boilerplate, test scaffolding, a first draft in an unfamiliar framework) gains the most. Work in a large codebase the developer already knows well, where the hard part is judgment, gains the least, and that is exactly what METR tested in 2025.

Where AI clearly helps, and where it clearly hurts

The studies above measure averages. The clearest wins and losses come from specific cases, and both kinds are on the public record.

The best-known controlled win is narrow and well defined. In a 2023 experiment by GitHub and Microsoft researchers (Peng and colleagues), developers asked to build an HTTP server in JavaScript finished 55.8% faster with GitHub Copilot than a control group without it. That’s a greenfield task with a clear spec, the kind of work the model above predicts AI helps most.

The biggest reported win at scale is the same shape. In August 2024, Amazon CEO Andy Jassy said Amazon Q’s code transformation feature cut the average time to upgrade an application to Java 17 from about 50 developer-days to a few hours, and estimated the saving at 4,500 developer-years and $260 million a year. Amazon also said its developers shipped 79% of the auto-generated code reviews without further changes. These are Amazon’s own figures about Amazon’s own product, so treat them as a vendor claim. The task type is still telling: a large, repetitive, rule-driven migration.

The best-known failure is the opposite shape. In July 2025, SaaStr founder Jason Lemkin reported that Replit’s AI agent ran destructive commands against his app’s live database during a declared code freeze, deleting records for more than 1,200 executives, and then told him a rollback would not work. Replit’s CEO called it unacceptable and said Replit would separate development and production databases automatically. The lesson is about access, not intelligence: the agent could reach production, and nothing stopped it.

Security sits between those. Veracode’s 45% figure means AI-written code that touches authentication, payments or data access needs the most careful review, and that’s also where the Supabase data leaks in AI-built apps have come from.

A rule follows from all of this. Use AI freely where the task is well specified and the result is easy to check, review AI-written code at least as hard as human code, and never give an agent direct write access to production.

We applied that rule while writing this blog. For our post on running OCPP 1.6 and 2.0.1 in one backend, we checked every protocol claim against the official JSON schemas and ran the code against simulated chargers before publishing. That caught two things a first draft got wrong: an assumption about how the WebSocket library handles version mismatches, and meter readings arriving as strings in OCPP 1.6. While researching this post, the same habit caught a stale number: METR’s own page now says its 2025 result is out of date, which a summary we started from didn’t mention.

What to ask a team that says AI makes them fast

If you’re hiring a development team and speed is part of the pitch, these questions separate a real process from a slogan:

1. How do you review AI-written code? Listen for a specific rule, not “we review everything”.
2. What do you test before merging? DORA’s finding means AI without tests produces more unstable releases.
3. Where do you not use AI? A team that has thought about it will have an answer, usually around security, payments or data access.
4. How do you check security? Given Veracode’s 45%, ask what scanning or review catches the common flaws.
5. What has AI gotten wrong for you? Every team using it seriously has examples. A team with none isn’t looking.

Takeaways

1. Feeling faster and being faster are different, and the gap has been measured. Trust delivery data over impressions, including your own.
2. AI amplifies the process around it. Tests, small changes and real review decide whether more code means more value or more incidents.
3. Treat AI-written code as untrusted until reviewed, especially for security. Bigger models haven’t closed that gap.

FAQ

Does AI make developers faster?

The evidence is mixed. METR’s randomized trial found experienced developers 19% slower with early-2025 tools. Its February 2026 update estimated a speedup with late-2025 tools but called the data unreliable because developers refused to work without AI. Surveys show most developers believe AI makes them more productive.

What was the METR AI productivity study?

A 2025 randomized controlled trial by METR. Sixteen experienced open-source developers completed 246 real tasks in large repositories they knew well, with AI allowed or disallowed at random. Tasks with AI took 19% longer, though developers believed they were 20% faster.

Is AI-generated code secure?

Often not without review. Veracode’s July 2025 report found security vulnerabilities in 45% of AI-generated code across more than 100 models, and larger models did not perform significantly better.

What does DORA say about AI in software development?

DORA’s 2025 report found AI adoption linked to higher delivery throughput but also rising delivery instability. It describes AI as an amplifier of an organization’s existing strengths and weaknesses.

Do developers trust AI coding tools?

Less than they use them. In the Stack Overflow 2025 survey, 84% use or plan to use AI tools, but 46% distrust the accuracy of the output and 33% trust it.