Velocity is only useful as long as yesterday's throughput remains a good predictor of tomorrow's delivery. AI changes the assumptions underneath that model.
In one Rootstrap engagement that transitioned to our AI-native delivery model, story points delivered per sprint increased 188%, from 17.8 to 51.3, while the engineering team went from three developers to two. By traditional output measures, productivity appears to have exploded. But it also exposes the limitation of the metrics.
When execution gets cheaper and faster, evidence of successful delivery remains what matters, while evidence of production becomes an increasingly poor indicator of progress.
Key takeaways
- Velocity measures one team's completion rate of its own units of self-assessed relative size, within one time box. It was never a measure of productivity, value, or delivery quality.
- More output is evidence of increased execution capacity, not automatically increased value.
- Output, delivery, and outcome are three different things, and AI affects them very differently.
- No single number tells you whether a software team is performing well. Balance delivery with stability, and connect both to business impact.
- Moving from stories to capabilities does not mean replacing story points per sprint with capabilities per week.
What does velocity really measure?
Velocity measures one team's completion rate of its own units of self-assessed relative size, within one time box. It is a planning input, calibrated locally, that only means anything inside the team that produced it.
It was never a measure of developer productivity, value, or delivery quality. In our own framework, it never had a target.
Three assumptions had to hold for velocity to be useful for us:
- Story points remain reasonably consistent as a relative sizing unit within the team.
- Team capacity per sprint stabilizes as the team matures, so past completion predicts future completion.
- Completed work is a reasonable proxy for progress, so counting finished items tells us something about what is left.
Under those conditions, velocity helped us answer a useful question: how much of this backlog will this team get through in the next few iterations?
Our own old metrics framework put the rationale plainly: velocity is the average amount of work a scrum team completes, measured in story points, and it is very useful for forecasting. The more iterations, the more accurate the forecast.
Story points, sprint capacity, and velocity were developed in a world where software execution was tightly constrained by human delivery capacity. The amount of work a team could produce in a sprint was a useful, if imperfect, signal for how quickly it could move through a backlog.
AI changes that relationship because execution capacity is no longer determined primarily by the number of engineers and the amount of human implementation time available in a sprint. Agents can take on implementation work, execute tasks in parallel, and dramatically change how much can be delivered.
That matters because velocity is only useful as a forecasting signal as long as yesterday's throughput remains a reasonable predictor of tomorrow's delivery. If AI changes the execution capacity underneath the metric, the problem is not simply that velocity becomes "bad." Its forecasting signal becomes less reliable.
And that is the real reason we need a different model. We still need to forecast delivery. We just need a unit of prediction that stays meaningful when the way software gets produced changes.
What happens when AI allows teams to produce more tickets or code?
Output goes up, sometimes dramatically. But the relationship between output and progress becomes harder to interpret.
Story points, pull requests, and lines of code can all keep moving in the "right" direction. The numbers are real. What changes is what we can infer from them.
A team producing three times as many story points, pull requests, or lines of code is producing more implementation output. But that does not tell us whether it is delivering the right capabilities faster, getting them accepted into production with less rework, maintaining quality and stability, or creating more value for the product.
More output is evidence of greater production. It is not, by itself, evidence of better delivery.
We saw this happen in one of our own projects
One Rootstrap engagement involving a consumer health and wellness mobile app gives us a particularly useful before-and-after comparison, because the project transitioned from our previous delivery approach to our AI-native delivery model. We compared five two-week sprints under the previous approach with three two-week sprints under AI-native delivery, using the same repositories and normalizing by day where necessary:
- Average story points delivered per sprint increased 188%, from 17.8 to 51.3.
- Lines added per day increased 87%, from approximately 985 to 1,840.
- At the same time, the engineering team went from three developers to two.
Same project, before and after the move to AI-native delivery. Output rose on every measure while the team went from three developers to two. Per-day figures are normalized. Source: Rootstrap engagement data.
By traditional output measures, productivity appears to have exploded. Story-point velocity almost tripled while the team became smaller. That is an important signal: clearly, something fundamental changed in the team's ability to execute.
But it also exposes the limitation of the metrics. The original forecast assumed a velocity of 24, our best estimate at the time based on historical delivery experience. Actual velocity during the full engagement averaged 35, and in the later phase it reached roughly 51. The historical relationship between team size, story points, and elapsed time that had informed that forecast was no longer holding.
The team had changed the system of production, but we were still measuring it with units calibrated against the previous system.
That leads to a few important lessons.
- More output is evidence of increased execution capacity, not automatically increased value. The increases in story points and lines added tell us the team was producing substantially more. They don't tell us whether the product delivered 188% more value, whether quality improved, or whether the right things reached production.
- When the production system changes, historical velocity becomes a weaker forecasting baseline. A velocity assumption was useful only while the relationship between the team's capacity, its sizing system, and its execution model stayed reasonably stable. Once AI materially changed that execution model, historical velocity was no longer a reliable predictor of delivery under the new model.
- The work is moving up a level of abstraction. In our AI-native approach, the capability becomes the primary human-facing unit. A capability is a complete piece of product behavior that humans define, validate, and accept as a whole, while agents execute the implementation underneath. That is where we define what needs to exist, why it matters, how it should behave, the technical design and constraints that should guide its implementation, and the evidence required to validate it. The implementation underneath is decomposed into agent-facing specifications and executed with AI. This shifts where human judgment is most valuable: away from managing individual implementation tasks, and toward designing the right solution, setting the right technical boundaries, and validating the capability as a whole.
- The increase in execution volume makes operating at the old level of granularity impractical. AI can multiply implementation artifacts much faster than it can multiply human attention. The answer is not to make engineers manage and inspect more artifacts faster. It is to move the human control point up a level, to capabilities and their evidence rather than the volume of artifacts produced underneath them.
Lines of code tell us how much code was generated, not how much engineering progress was made. Story points depend on a human-effort-based unit whose relationship with delivery becomes increasingly unstable when agents perform a significant share of the execution. Neither gives us a useful basis for understanding the progress of an AI-native team.
Why does more output not necessarily mean more value?
Because output, delivery, and outcome are three different things, and AI affects them very differently.
AI has made producing software dramatically cheaper and faster. Code, tests, implementation tasks, and iterations that once consumed significant engineering time can increasingly be generated or executed by agents.
But producing code was never the end goal. The software still needs to be correct, accepted, reliable in production, and useful to the people it was built for. AI can help across those stages too, but increasing the volume of implementation does not guarantee any of them.
In the engagement described above, the numbers clearly tell us our execution capacity changed dramatically. They cannot tell us whether we built the right things, whether those changes worked as intended in production, how much rework they generated, or whether they produced the expected product outcome.
Those are different questions, and they require different evidence.
Output, delivery, and outcome answer different questions. As AI makes output cheaper, delivery and outcome carry more of the evidence that a team is performing.
Output: how much did we produce? Traditional volume indicators include PRs, commits, lines of code, tickets, and story points. As AI makes implementation faster and cheaper, these measures become increasingly weak proxies for actual delivery progress or value.
Delivery: how reliably do we turn validated intent into working production software? Signals include capability cycle time, rework, defects, forecast calibration, etc. Increased execution capacity only represents an improvement when it translates into reliable, accepted software, without simply shifting costs into verification, rework, or production instability.
Outcome: did the software accomplish what we built it to accomplish? Relevant signals include adoption, business impact, user behavior, retention, and operational improvement. These help establish whether successful delivery translated into the intended value, although outcomes may take longer to materialize and can be influenced by factors beyond engineering.
This distinction matters because an AI-native team can look extraordinarily productive at the output layer while performing poorly at the other two. An AI-native team can accelerate implementation while introducing more defects, increasing rework, or creating instability in production. It can deliver capabilities that meet their technical requirements but fail to achieve the intended business outcomes. None of those differences are visible if the measurement system stops at output.
The objective of measuring delivery performance has not changed. We have always needed to understand whether teams are delivering reliably and creating value. What AI changes is how little we can infer about those results from implementation volume alone.
That makes two questions increasingly important:
How reliably are we turning validated intent into working software?
Is that software producing the outcome we intended?
AI doesn't make measurement less important. It makes it harder to justify using output as a shortcut for the things we actually care about.
Which engineering metrics should matter in an AI-native team?
We measure AI-native teams across four dimensions: Delivery Performance, Delivery Quality & Stability, Ground Truth Health, and Project Health & Client Happiness. The organizing principle of our current metrics framework is simple:
No single number tells you whether a software team is performing well. Balance delivery with stability, and connect both to business impact.
The questions we need to answer have not fundamentally changed. We still want to know whether a project is delivering predictably, producing quality software, maintaining a healthy team and client relationship, and ultimately creating value. What changed is which signals can answer those questions reliably.
Our previous framework relied heavily on metrics built around human execution: velocity, sprint burndown, Say-to-Done, and Time to Merge. As our delivery model changed, those signals either needed to be replaced or reinterpreted.
We're evolving our delivery metrics framework around four dimensions: Delivery Performance, Delivery Quality & Stability, Ground Truth Health, and Project Health & Client Happiness. They are not a ranking or a menu. A project has to be read across all four. Faster delivery means little if rework is increasing; excellent engineering quality means little if we are building against the wrong requirements; and neither matters much if the product fails to create value for the client.
Rootstrap’s four metric dimensions for AI-native teams. A project is read across all four together, not ranked by any single one.
1. Delivery Performance: can we turn intent into working software predictably and effectively?
The goal is no longer to measure how much work humans complete inside a sprint. It is to understand how effectively and predictably the human + agent delivery system turns validated intent into operational software. The signals we are exploring include:
- Capability Cycle Time: how long capabilities take to move through the delivery system.
- Deployment Frequency: whether completed change is flowing into production regularly rather than accumulating in large batches.
- Capabilities Progress: how much of the agreed product scope has moved through delivery and how much remains.
- Scope Rework %: how much delivery effort is being spent redoing previously defined work.
- Scope Change %: how much the target itself changes during execution, which helps us distinguish execution problems from evolving business intent.
The distinction around capabilities is important. Moving the human unit of work from stories to capabilities does not mean that "capabilities per week" becomes our new velocity. Capabilities give us a better level at which to define, validate, and observe product progress. They are not a new unit for measuring engineering output.
For forecasting, we care about how long comparable capabilities historically take to become operational, not how many capabilities a team can produce in a time box.
How we're rethinking forecasting
Our goal is to move from effort-based estimation toward evidence-based forecasting. Instead of asking engineers to assign effort estimates and using team velocity to translate them into delivery timelines, we're working toward a model where capabilities are classified by tier × size and forecasts are informed by how long comparable capabilities have actually taken to deliver.
We're already building the evidence base for this approach. As of October 13, 2026, our dataset included 161 delivered capabilities across 15 projects, providing a growing body of observed delivery data that we can use to understand how different classes of capabilities behave.
The goal is not to find a new fixed average or assume that historical observations will always predict future delivery. It's to build a forecasting model that continuously incorporates new evidence, accounts for differences between types of work, and makes uncertainty explicit through probability distributions such as P50, P80, and P95.
Just as importantly, we want to measure whether those forecasts deserve to be trusted. If we assign a P80 forecast, we should eventually be able to verify whether approximately 80% of comparable capabilities were delivered within that bound.
That will give us something velocity never directly provided: a forecast, and a measurement of how much we should trust that forecast. As more comparable capabilities are completed, the historical distribution gets richer. The measurement system learns from delivery itself.
2. Delivery Quality & Stability: did what we delivered actually work?
Higher execution capacity only helps if what reaches production is correct, maintainable, secure, and behaves as intended. This dimension asks whether increased delivery is being achieved without transferring the cost downstream into defects, instability, or excessive human verification. We look at signals such as:
- Eval Coverage %: whether capabilities have the behavioral verification appropriate for their class, making evidence part of the definition of quality rather than an after-the-fact check.
- Defect Escape Rate: how often defects escape the delivery process and reach later environments or production. Importantly, this stays useful regardless of whether the code was written by a human or an agent.
- Maintainability & Reliability: whether faster production is degrading the long-term health and operability of the system.
- Vulnerability Remediation SLA % and Security Rating: whether increased speed is being achieved without accumulating security exposure.
- Hotfix Rate: whether apparently completed delivery is creating unplanned corrective work immediately afterward.
This is an important counterweight to speed. A cycle time that falls while escaped defects or hotfixes rise is not automatically an improvement.
That direction is consistent with the broader industry's approach to measuring software delivery. DORA has long emphasized balancing throughput with stability, and its introduction of deployment rework rate in 2024 made the cost of corrective work even more explicit. The principle remains the same: faster delivery is only meaningful when it is accompanied by reliability and stability, rather than increased failures or rework.
3. Ground Truth Health: are we building from reliable intent?
This is the genuinely new dimension in our framework. AI makes execution cheaper, but that increases the leverage, and the risk, of everything that happens before execution begins.
An agent can implement a well-defined capability faster. It can also implement the wrong capability faster. That makes the quality of the context feeding the delivery system something we need to manage, validate, and measure explicitly.
We call that context Ground Truth: the validated business model, decisions, requirements, constraints, and knowledge that humans and agents rely on to build. We currently look at three signals:
- GT Capability Readiness % (complete): how many capabilities enter implementation with high confidence and without unresolved questions. This is particularly important because it measures risk before we pay the cost of building.
- GT Freshness (current): whether open business questions and decisions are being resolved at roughly the rate new ones emerge, or whether ambiguity is accumulating.
- GT-Attributed Rework % (trustworthy): how much work had to be redone because the underlying business model, decision, or context was wrong, rather than because of an implementation defect or an intentional scope change.
4. Project Health & Client Happiness: is the delivery system sustainable and creating confidence?
Software delivery is still a socio-technical and commercial system. A project can have excellent cycle times and clean production metrics and still be unhealthy, because the team is burning out, the budget is drifting, or the client has lost confidence in what is being delivered. So we continue to look at:
- Budget Deviation: whether actual spend is tracking against the commercial plan.
- Project Team Satisfaction: whether the people operating and supervising the system can do so sustainably and effectively.
- Client Sentiment: whether the client sees progress, trusts the delivery process, and is satisfied with both the results and the service.
AI does not make these signals obsolete. If anything, it makes them useful checks against optimizing the delivery system in isolation from the people and business it exists to serve.
The principles stay the same. The signals change.
The principle of reading metrics as a system is not new for us. Our previous framework already recognized that no single KPI could describe the health of a project: delivery speed had to be understood alongside quality, team health, client satisfaction, and business context.
What AI changes is not that principle, but the signals underneath it and how we interpret them. Velocity and other execution-oriented metrics made sense when human delivery capacity was the dominant constraint. In an AI-native delivery system, we still need a balanced view. But that view now has to account for the quality of the intent entering the system, the behavior of human + agent execution, the reliability of what reaches production, and the evidence behind our forecasts.
A few principles become particularly important:
1. Read flow together with stability and rework. This was always important; AI makes the relationship more consequential because execution can accelerate much faster. A lower Capability Cycle Time is only a positive signal if quality remains healthy. If Scope Rework, Defect Escape Rate, or Hotfix Rate rises at the same time, we may simply be moving cost downstream.
2. Measure whether our predictions deserve to be trusted. A forecast is not useful simply because we can produce a number. Calibration makes the reliability of the forecasting system itself measurable, and allows it to improve as more evidence accumulates.
3. Don't turn the new units into new output targets. Moving from stories to capabilities does not mean replacing story points per sprint with capabilities per week. Otherwise, we would reproduce the same problem with a different vocabulary.
We are moving away from signals centered on how much human work gets produced, and toward signals that tell us whether the right intent entered the system, how predictably humans and agents turned it into working software, how much correction that required, whether the result can be trusted, and whether the project is creating the intended value.
The question was never simply:
How much did we get done this sprint?
The broader question has always been:
Is this project healthy and delivering successfully?
What AI changes is the evidence we need in order to answer it.
