AI is making it easier than ever for engineering teams to produce code. That does not automatically mean they are shipping more value. In many teams, the real constraint has simply moved.
Code now arrives faster than review queues, test pipelines, approval paths, and release processes can absorb it.
That is why engineering leaders need to rethink what they measure. The question is no longer just whether teams are coding faster. It is whether faster generation is improving delivery capacity, reducing risk, and turning into production outcomes.
A practical CTO scorecard to start with
If you are starting out, keep the scorecard small. Use a few measures that show whether AI speed is turning into actual delivery outcomes. If some of these feel unfamiliar, the rest of the article explains why they matter.
Start with:
- Lead time for changes
- Pull request (PR) review time
- Change failure rate
- Recovery time
- Rework after review
- Pipeline duration
- Manual intervention frequency
Teams can adjust this set depending on context, but these measures balance delivery flow, quality, and control.
What changes in engineering metrics with AI-assisted development
Engineering outcomes still matter, but the path to those outcomes changes.
In a traditional model, most effort sits in writing and refining code. With AI-assisted development, drafting becomes faster while delay and risk shift into review, testing, and release stages.
Dashboards may still look healthy. But delay and risk are now showing up in different parts of the system.
The DevOps Research and Assessment (DORA) metrics remain a strong outcome framework. Deployment frequency, lead time for changes, change failure rate, recovery time, and reliability still matter because they show whether software is delivered safely and consistently.
What changes with AI is not these metrics themselves, but what drives them. Faster code generation shifts pressure to review, testing, and release stages. DORA alone may not explain where problems are building up.
This matters because hidden bottlenecks are not just an engineering inconvenience. They affect release predictability, incident exposure, team capacity, and the cost of delivery.
A team may appear more productive because draft output rises, while reviewers are overloaded, testing cycles get longer, and more work waits for release approval. If leaders do not measure these signals, they can end up funding speed in one part of the system while creating delay and instability everywhere else.
That is why leaders need measures that show where delivery is slowing down, not just where code is being generated faster.
Traditional coding metrics that still matter in AI-assisted development
Sprint velocity
In traditional coding, velocity is a rough proxy for team output. With AI-assisted development, it becomes harder to interpret because the same point count can come from very different levels of human effort.
The number can stay the same while the meaning changes.
Cumulative Flow Diagram (CFD)
In traditional coding, cumulative flow diagrams help teams spot bottlenecks in the delivery process. In AI-assisted development, they become even more useful because they show where work starts piling up once drafting gets faster.
If coding speeds up and review, QA, approval, or release stages stay slow, the backlog will show up there first.
Delivery plan / predictability
In traditional coding, predictability mainly reflects how well a team estimates and executes its work. As teams adopt AI-assisted development, they can feel faster without improving actual delivery.
Tracking commitment reliability, release predictability, and change completion patterns helps leaders see whether the system is becoming more dependable or simply producing drafts faster.
Getting changes safely shipped is now the harder part
For many teams, AI makes code much faster to create. Review capacity, test quality, pipeline speed, and release confidence often decide how much of that speed turns into real delivery gains.
If review, validation, and release stay slow, faster generation will not improve delivery on its own. This is where leaders can misread progress. Drafting speed rises first. Delivery improves only when the rest of the system can absorb the added volume.
That is why teams adopting AI should pay close attention to a few operational measures:
- Pull request (PR) review time and time to merge: If AI increases output but review time keeps rising, reviewer capacity, trust, or code quality is the real bottleneck.
- Build and pipeline duration: Faster generation only matters when validation, security checks, and release automation can keep pace.
- Rework rate or code churn: Measure the amount of change after initial review, such as lines modified or commits added. A useful check is whether generation time plus rework time is lower than traditional development for similar work.
A common pattern in practice
Consider a team that introduces AI coding tools across a product squad and quickly sees more pull requests opened each week. At first, this looks like a productivity win. But within a few sprints, review queues grow, QA validation takes longer, and more changes come back for rework after security or architecture review. The team is generating code faster, but release throughput has not improved much. In some cases, it gets less predictable.
This is the pattern leaders need to watch for. Early gains in drafting do not automatically translate into better delivery. Review capacity, test automation, and release controls still determine how much value reaches production.
Metrics that now mislead more than they help
Some older metrics are now weak signals at best and misleading at worst.
Recent industry examples show why raw AI usage can become a misleading signal. Amazon removed an internal AI leaderboard after employees inflated token consumption in ways that raised costs without reflecting useful delivery progress. Reporting says the company shifted toward deployment-oriented measures that better reflect real code shipping and business value. The lesson is simple: if a metric rewards activity without tying it to outcomes, teams eventually learn to optimize for the metric instead of the mission.
- Lines of code: AI can generate a lot of code quickly. More code is often a sign of more complexity, not more value.
- Story points as a productivity measure: If teams keep the same estimation scale but complete more work using AI, output does increase. But the signal becomes harder to interpret because effort per story may drop. Without recalibration, story points reflect output volume rather than effort or efficiency.
- Hours spent coding: Time in the editor is a poor proxy for value, especially when human effort shifts toward prompt design, review, and decision-making.
- Token consumption: High token usage can reflect longer prompts, retries, or inefficient tool behavior rather than better engineering outcomes. It is easy to inflate, expensive to reward, and weak as a business metric unless tied to shipped changes, quality, or cycle time improvements.
What to add when the team moves from traditional coding to AI-assisted development
As teams adopt AI-assisted development, the scorecard needs a few extra measures. A practical next step is to add measures for the three common forms of AI-assisted development: AI-assisted coding, vibe coding, and agentic coding.
For AI-assisted coding
AI-assisted coding usually improves local implementation speed first. The main question is whether that faster drafting is actually reducing delivery friction.
- Suggestion acceptance rate: Use this only as an adoption signal. Accepted suggestions are often edited or replaced later, so this metric does not reflect quality or value on its own. Always pair it with downstream signals such as rework, retained code, or PR cycle time.
- Time to first review: This reveals whether faster code creation is simply pushing work into review queues.
For vibe coding
Vibe coding is useful for fast exploration, UI spikes, internal admin tools, and early product ideas, especially when teams want to test an idea quickly. But when teams rely on code they have not meaningfully reviewed, maintainability, correctness, and security risks rise quickly. The real question for leaders is whether that output can be cleaned up, reviewed, and shipped without heavy rework.
- Prototype-to-production conversion rate: If many vibe-coded outputs never reach production, treat them as experimentation capacity and track delivery throughput separately.
- Rework after review: High rework usually signals maintainability gaps, weak architecture fit, or shallow requirement alignment.
For agentic coding
Agentic coding can help with refactoring, test generation, migrations, and other repetitive work. It also needs stronger guardrails because the tool is taking more action on its own.
- Manual intervention frequency: Track how often humans need to stop, redirect, or correct an agent's path.
- Rollback or revert rate: If agent-executed changes are reversed often, the system is moving faster than your controls can support.
Where these modes fit best
AI coding modes fit different contexts. In more critical systems, metrics should emphasize review discipline, test strength, traceability, and rollback readiness.
- Vibe coding fits best: Prototypes, internal tools, UI spikes, and low-risk workflows where speed matters more than long-term structure.
- Agentic coding needs stronger controls: Use tighter review and rollback standards in regulated, security-sensitive, or legacy-heavy environments.
Keep developer experience in the scorecard
AI changes where engineering effort is spent across the workflow.
- Flow: The amount of uninterrupted time developers spend moving work forward without waiting on reviews, clarifications, or rework. A drop in flow usually means more interruptions or delays in the system.
- Satisfaction: Do developers feel AI tools are genuinely useful in daily work?
Developers will spend more time prompting, reviewing, and steering AI. Leaders need to know whether that effort improves outcomes or simply adds overhead.
Closing thought
AI has changed the economics of writing code. It has not changed the need to ship safely, predictably, and at the right pace for the business. The teams that benefit most will not be the ones generating the most code. They will be the ones measuring whether that code actually makes it through the system and into production with control.
References
- DORA. DORA’s software delivery performance metrics.
- DORA. Accelerate State of DevOps Report 2024.
- DORA. Accelerate State of DevOps Report 2021.
- GitHub Blog. Measuring GitHub Copilot’s impact on engineering experience and productivity.
- Martin Fowler. Vibe Coding.
- InfoWorld. Amazon deletes devs’ tokenmaxxing leaderboard to minimize costs.
- Datadog. Measure the real impact of AI coding tools on software delivery with Datadog AI Impact.
