Agents Will Get Things Wrong: How to Plan for AI Agent Mistakes
AI agents will get things wrong, and sometimes they’ll do it with a lot of confidence. Production-grade AI delivery doesn’t pretend otherwise: it assumes the agent will be wrong, checks every result against criteria agreed before the build, keeps evidence of what was checked, puts people where judgment is needed, and changes whatever caused each failure.
I’m Alicia Olocco, QA Manager at Rootstrap, and most of my team's work now happens around agents. The biggest lesson so far: generating code is no longer the hard part. Knowing whether that code is right is.
How to plan for AI agent mistakes
- Agree on acceptance criteria before anything is built.
- Accept only what you’ve seen working, with evidence.
- Run evals with real and edge-case data.
- Require a person to decide and sign off on security, permissions, payments and business rules.
- Trace every result to its spec, its context and the agent that produced it.
- Fix the cause, not just the output.
How has QA changed now that agents do most of the execution?
QA now spends its time on what agents can't decide: whether the requirements are right, what's critical and where the risk is. QA at Rootstrap was never just the last step, since we've always worked from the start of each project. What didn't change for a long time was the execution: test cases written ticket by ticket, run once the code was ready. When agents sped up development, that model stopped scaling. On some projects, output grew faster than any traditional QA process could validate, and adding people would only have postponed the problem.
Today my team spends very little time writing test cases or running manual regressions, because agents handle most of that. We spend much more time validating acceptance criteria before anything is built, reviewing what the agents produce and doing exploratory testing where the risk is highest.
What does an eval engineer do in QA?
An eval engineer measures how well an AI system works and whether it works consistently, not just whether it passes once. AI evals are tests that measure how well an AI system’s outputs meet defined criteria across many realistic and edge cases, instead of checking a single expected answer. In traditional testing, the same input always gives the same output. With AI, the same input can give different answers, and more than one may be acceptable.
So QA now builds datasets with real, edge and adversarial cases, defines what a good result looks like, and reruns those evaluations every time a prompt, a model or the agent's context changes. That takes technical skill to design evaluations and product knowledge to know what's worth evaluating. You can't test well what you don't understand.
For me as a manager, this changes who I look for and how I grow my team. Running tests is no longer the core skill. Knowing the product, spotting what nobody defined, and designing how to evaluate a system that doesn’t always behave the same way are.
Why do AI agents make mistakes?
AI agents make mistakes for two reasons: the context they’re given and the model itself. In Rootstrap’s experience delivering AI-native projects, context causes many of the most costly ones: missing or outdated information, unclear requirements, or edge cases nobody defined. When something is ambiguous, the agent fills the gap with an assumption instead of asking. Those causes can be reduced before building, even if some only show up with real data. What we can’t eliminate is the model’s own tendency to be confidently wrong or to invent things, known as hallucination. We can choose models carefully and constrain them, but that part we can only catch.
The pattern we see most is confidence without evidence. Agents present guesses as facts, like listing several causes for a bug as certain without checking any of them. When we switched AI providers on one project, we ran the same 24 real documents through two models, twice each. One read a $1,379 payment as $379. The other, in one of its two runs, hallucinated an invoice that didn’t exist, taking the figure from a table of outstanding balances.
Agents also do exactly what's written and nothing more, skipping edge cases nobody asked for, and they mark things as done too easily: tests that "partially passed" while hiding bugs, or fixes marked as resolved that were still broken.
The easy mistakes are the ones that break loudly: errors, failing tests, flows that don't finish. The hard ones look right: a believable number that's wrong, invented data in a perfect format, or something that works but solves the wrong problem.
Spectrum of AI agent mistakes, from easy to catch to hard to catch
Why does an AI demo break in production?
An AI demo only has to work once, on a good path, with clean data. Production has to work every day, with real users who do unexpected things, integrations that fail and a product that keeps changing. And because agents don't always give the same answer, something that worked in a demo can fail the next time.
We saw this on a platform that supports auditors, where the AI turned uploaded documents into invoices and expense amounts. Some results were off by large amounts, with real financial impact. In one case, the AI added more than $30,000 to an invoice by counting sub-item totals that were already included in the overall total. The main issue wasn’t the model: clean demo data never had documents like these. The same happened with proofs of payment: when one included a copy of the invoice, the system created a second invoice for the same expense. Combined documents like these only showed up once real data came in. Together with the client, we redefined the behavior so proofs of payment are no longer analyzed by the AI, verified that one document now produces exactly one invoice, and added evals for those cases.
How are AI agent mistakes different from human mistakes?
AI agent mistakes differ from human ones in two ways. When information is missing, an agent fills the gap instead of stopping to ask, unless the system makes it stop. And its mistakes are systematic: the same one repeats across many outputs. That’s riskier at scale, but it also means that fixing the cause once fixes it everywhere.
How do you verify AI agent output?
Verifying AI agent output means checking it against acceptance criteria agreed before the work started, and only accepting what you’ve actually seen working. An agent saying "done" isn't enough. Those criteria live in the Ground Truth, the project's shared source of what the product should do and why: capabilities and user stories, business rules, decisions and who made them, integrations, and how the current system behaves. Agents help build it, but a person reviews and approves it. When something is missing or unclear, the agent doesn't guess. It becomes an open question for a person, and nothing gets built until it's answered.
After a task is finished, a QA agent tests the application the way a user would and marks each criterion as met, partial, not met or blocked, with evidence for each. Every finding is classified first, because not everything is a bug: some are inconsistencies, spec gaps or environment issues, and only real defects go back to development. One agent fixes the issue and a different one re-tests it. When all stories are done, we test the whole capability again, because things that work separately can break together.
Diagram of how QA verifies AI agent output before production
At Rootstrap, “right” means three things, in this order: it meets the acceptance criteria, it doesn't break anything else, and it meets our quality standards. A green pipeline isn't proof that something works for the user.
Knowing exactly what you're testing matters more than it seems. On one mobile project, features that had passed every test on the web were next in line for the iPhone app. A version check we added as a precaution showed the app on the simulator was weeks old, so the agent stopped and reported it instead of testing anyway. With the right version installed, we found three real bugs that never showed up on the web, including a form that lost what the user had typed. Without that check, every result would have been false.
When should a person step in on an agent's work?
A person steps in on an agent’s work when something isn’t defined, needs approval, keeps failing, needs real human testing, or can only be done by a person, like granting access or setting up real accounts. At Rootstrap, clear rules decide when a person steps in. What that person brings is judgment. If a requirement is missing, the agent stops and asks whoever owns the decision. The QA engineer signs off on every verdict, because someone has to own what ships. If a fix bounces back twice, the agent escalates, because by then the problem is usually the spec or the environment. Every intervention is recorded with the person’s name. In practice, people decide what nobody has defined and own what ships, while agents do the checking. Matías Mansilla, our VP of Engineering, explains the thinking behind this in human in the loop: decide, own, or check.
What should you never leave to an AI agent without human review?
Security, permissions, payments, and any change to a business rule always need human review. In these areas a mistake is expensive and often silent, and someone has to own the decision and be able to explain it to the client.
That’s also where we test most deliberately. On one project, we had agents test every user role and every payment path, including combinations nobody had written down. That surfaced issues no error log would have shown, like a permission that let a user do more than their role allowed, or a charge that shouldn’t have happened. The agents gave us the coverage. Deciding how serious each issue was and what to do about it was a human call.
Agents also never handle real credentials or real customer data, only test data. Giving an agent more access than it needs is one of the top risks OWASP lists for AI applications.
How do you balance speed and control with AI agents?
Balance speed and control by drawing the line by risk, not by volume. Agents do the checking everywhere. In critical areas, a person also decides and signs off, while lower-risk changes move forward on automated verification alone. Every time the system asks a person for something, we record whether that person changed anything, but a review that's almost always approved isn't removed automatically. A check that rarely catches something can be the one that catches the rare, expensive mistake.
The line has moved both ways. We got stricter where it mattered: a partial pass no longer counts as a pass. We loosened it where a check was blocking more than it should: a control that’s too strict ends up being bypassed, so each check now blocks only the work it actually affects.
What evidence should an AI agent’s work leave behind?
Every piece of an AI agent’s work should be traceable from the request to the final result: what was asked, which version of the spec was used, what context the agent had, what it produced and how it was tested. Every commit, pull request and bug shows which agent created it, and every approval has a person’s name on it. Nothing gets overwritten, and if part of the trail is missing, it shows as missing.
A traceable record of agent work pays off every day. When a regression test fails, we classify the cause from the evidence before anyone reruns it, which puts an end to “let’s run it again and see if it passes.” On the same auditor platform, over roughly one sprint, the records showed that several bugs had been marked as done, so they went back into the re-test. Three of seven were still broken. Without that trail, they would have been closed.
For clients, this changes how quality gets proven. Each delivery ends with a complete report of what was tested, what passed, what failed and what’s still open, with every bug linked to the test case that found it.
Does “created” mean “correct” when an agent writes to another tool?
No. When an agent creates something in another tool, like a ticket or a test case, what gets saved isn’t always what it sent. So we read it back and compare it with what a person approved. On a mobile wellness app, an agent reported every bug as created in our tracker, but some tickets were missing fields, and one never appeared on the active board.
The agent’s confirmation looked exactly the same in every case. Only the QA lead re-reading the tickets showed the difference. Now the agent sets every field explicitly and re-reads each ticket right after creating it.
How do you keep an AI agent from making the same mistake twice?
To keep an AI agent from repeating a mistake, fix the output and then change whatever let the mistake happen. If the requirement was unclear, we update the Ground Truth. If the agent was missing context or misunderstood its task, we adjust its instructions. If no test would have caught it, we add one, and every bug found without a test case gets a regression test. If a check proved too weak or too strict, we change the process. Any change goes through the same review as code.
Learning loop showing how each AI agent mistake changes the system
Two failures on a mobile wellness app show how changing the agent, not just the output, prevents repeat mistakes. First, a spec-writing agent wrote a layout criterion from a screenshot and the technical data of the design file, never from the design itself. The criterion contradicted the design from day one, and QA only caught it after the screen was built. So we changed the agent’s instructions: any criterion that describes a layout is now checked against its design before the build, and if they disagree, the story is blocked until a person decides which source is current. A first draft of that rule said “the design wins.” The QA lead rejected it, because designs go out of date too.
Second, the agent that designed test cases never specified what kind of user each test needed. Every test ran on the same shared test account, and that account happened to have a particular setup. So every test passed, but only for users with that profile. Nobody had tested users with a different setup, and that’s where a high-severity bug was hiding: for them, sessions wouldn’t start at all. QA found it outside the planned tests. We updated that agent so every test case now states the user conditions it depends on, like the account’s setup, the device language, or whether a list is empty or very long. That way, each type of user gets tested on purpose instead of by accident.
Lessons also carry over between projects, and that's the part I value most. A better way to test something gets generalized, with nothing specific to the client, and reviewed by people before it becomes part of Rootshift, our shared delivery system. Both rules above now apply to every project, so a new project starts where the previous ones left off.
Who tests the agents that do the testing?
The agents that test also need to be tested, because a verification agent you don’t evaluate is just another source of confident errors. Every bug and change is tagged with the agent and version that produced it, so we can see where a testing agent tends to fail, from missed duplicates to wrong severities and false positives, and fix that agent specifically.
Every run also records how many times work bounced back and why. Turning that into clear metrics is our next big priority, in the same spirit as the shift we describe in why sprint velocity is becoming the wrong metric for AI-native teams.
What I tell a client who's worried the AI will get things wrong
You’re right to worry, because it will. What matters is what happens around it.
We don't trust an agent because it says it's done. We check its work against what you approved before we started building, and we only accept what we've actually seen working. In critical areas, a person makes the call and signs off, and you don’t have to take our word for any of it: you get a report of what was tested, how and what was found. When something does go wrong, we find out why and change what caused it, so it doesn't happen again on your project or the next one.
Companies that speed up development without investing in clear specifications and solid verification end up shipping mistakes faster. And a feature that uses an LLM can pass every test with clean data and still fail with real data, which is why any AI feature needs evals with realistic cases from the start.
The real risk isn't that AI makes mistakes. It's working with a team that doesn't plan for them. If you want to see how we’d plan for AI agent mistakes on your project, start with how we work.
