One of the things I like about agentic testing is that the agent can be clever...
But, sometimes it's a little too clever!
Give an agent a goal and, much like a determined developer five minutes before a demo, it may find a way to make the thing work.
Traditional end-to-end tests do not usually have this problem. They tend to be deterministic: click this button, wait for this element, expect this text. If something unexpected happens, the test fails.
An agent approaches the same problem differently. Give it an objective and it can look around when a button is not where it expected, open a collapsed menu, wait through an unexpected delay, or find another way forward when something goes wrong.
That's pretty useful,
But it also creates an interesting problem:
A smart enough testing agent can make a broken application look like it works.
Mission success is not always application correctness
Imagine an agent testing an ecommerce application. Its mission is simple: add a product to the cart and verify that it appears there.
The agent clicks Add to Cart. Nothing happens.
A deterministic test might stop right there with a failure. But an agent has, oh so many, other options.
Maybe it inspects the page and notices that something seems to have happened behind the scenes. So it refreshes the browser. After the refresh, the product appears in the cart. Great. The agent continues to checkout and completes its mission.
Except we actually have a problem.
A normal customer should not need to know that refreshing the browser happens to reconcile the client state with the server. If the final report simply says PASS, then the intelligence of the testing system has hidden meaningful information about the application.
The agent did its job a little too well.
Passed after recovery
I've started treating this as a distinct test outcome in Testronaut: ⚠️ Passed after recovery.
A recovered mission means the agent ultimately accomplished its objective, but something happened along the way that required it to take an exceptional action to continue.
Or in other words: The mission passed, but the execution wasn't clean.
- Passed: the mission succeeded through a normal user journey.
- Passed after recovery: the mission succeeded, but the agent had to recover from an unexpected condition.
- Failed: the agent could not accomplish the mission.
That new middle state is where things get interesting.
So what exactly counts as a recovery?
There's a fuzzy line here, because clicking a menu isn't recovery. Scrolling down the page isn't recovery. Waiting half a second for React to do React things isn't recovery.
We're more interested in the moments where the agent effectively says, ‘Well crap, that didn't work, but I have another idea...’
A recovery is an action taken primarily to escape an unexpected application state rather than advance the intended user journey.
Another useful question is whether a typical user would reasonably be expected to know they should do this. If the answer is no, there is a good chance we have crossed into recovery territory (admittedly, this is a bit of grey area).
Refreshing or reloading the page
This is probably the clearest example. The agent performs an action, the UI becomes stale, incomplete, stuck, or inconsistent, and the agent reloads the page.
Developers know that a refresh is basically a minor orbital correction (excuse the pun). Something isn't quite where it's supposed to be, so you hit refresh and hope everything lines up again.
I don't expect users to typically do this.
A refresh can conceal stale client state, broken synchronization, rendering failures, routing problems, websocket failures, or any number of other defects. If refreshing fixes the problem, that's useful diagnostic information. It isn't a reason to pretend the problem didn't happen.
Backtracking and retrying navigation
Suppose the agent follows the apparent path to a feature and reaches a dead end. It goes back to the dashboard, reopens a menu, and tries another route.
That's good problem solving. It's also evidence. Repeated navigation, returning to an earlier page, reopening a route, or abandoning one path for another can indicate broken links, stale state, confusing information architecture, or inconsistent routing.
Repeating an action
Click. Nothing. Click again. Success. I'm glad the second click worked. I'd still very much like to know why there needed to be a second click.
The same applies to resubmitting a form, reopening a modal, toggling a control again, or repeating another action because the first attempt did not produce the expected result. Retries can be resilience. They can also be evidence.
Finding an alternate path
This is where agentic testing gets particularly interesting. Suppose the intended navigation does not work, but the agent finds another way to reach the same destination. Technically, the mission can continue.
But imagine the airlock is welded shut and the agent discovers it can crawl through an open maintenance hatch. I'm glad it got inside. I'd still like to know about the airlock.
If the primary Save button does not work, but pressing Enter submits the form, that's interesting. If a navigation link fails, but the agent discovers another visible route to the same screen, that's interesting. If the front door does not work, successfully finding the side entrance should not turn the front door green on the test report.
The workaround is part of the result.
Re-establishing lost state
Sometimes the agent may need to reopen a dialog, reselect an item, return to a previous screen, re-enter information, or otherwise reconstruct state that unexpectedly disappeared.
An intelligent agent may actually be unusually good at this. It remembers what it was doing, what it entered, and what the end goal is. A user may not have any of those advantages. The agent’s memory should not invisibly compensate for the application’s inability to preserve state.
Re-authenticating unexpectedly
If the mission starts authenticated and the agent suddenly encounters a login screen, it may be perfectly capable of signing in again. That is impressive resilience. It is also not normal.
Unexpected re-authentication could point to session expiry, authentication state synchronization problems, redirects, cookie issues, or another reliability problem. If the agent signs back in and carries on without mentioning it, then Houston, we have a problem.
Escaping unexpected UI
A modal appears that should not be there. An overlay blocks a button. A stale notification covers a control. The agent closes it, presses Escape, clicks outside it, or otherwise clears the obstruction and continues.
Again, good job, agent! But if dismissing that UI was not part of the intended journey, I want to know why it was necessary.
Waiting much longer than expected
Waiting is a little more nuanced. Modern applications are asynchronous. Waiting for something to happen is perfectly normal.
There is a difference, though, between allowing an operation to complete and realizing that nothing has happened, waiting significantly longer, and eventually seeing the expected result appear. The mission might technically pass. The user still spent fifteen seconds wondering whether the button did anything. I find that worth knowing!
Recovering from a failed interaction
Sometimes an interaction simply fails. A selector no longer matches. An expected element is not actionable. A click is intercepted. The page changes while the agent is interacting with it.
An agent may be able to inspect the new state, adjust its approach, and keep going. Some of these situations will be application defects. Some will be test problems. Some will just be the inherent messiness of interacting with a modern web application. The important thing is that the evidence survives.
Recovery doesn't necessarily mean bug
A recovered mission should not automatically mean BUG FOUND. The web is messy. Networks hiccup. Animations race. Third-party services respond slowly. Browsers do weird browser things. Test environments have bad days too.
The purpose of recording recovery isn't to have the agent diagnose every unexpected condition as a product defect. It's to preserve what happened.
Something unexpected happened here. This is what I encountered, and this is what I did to continue.
Then a human can decide whether that represents a bug, flaky infrastructure, harmless variability, or something else. That is considerably more useful than either failing everything unexpected or passing everything the agent can overcome.
Leaving footprints
I recently built a small demonstration of this into Testronaut. The mission begins on an authenticated dashboard and verifies that the expected controls are available. The demonstration then intentionally invalidates the current client view.
The agent knows how to solve this. It reloads the page. But before doing so, it records that a recovery is happening, why it is happening, and captures screenshot evidence of the state.
After the reload, the agent verifies the application again. Authentication persisted. The dashboard returned. The expected controls were usable. Mission accomplished.
But the report does not call that an ordinary pass. It calls it ⚠️ Passed after recovery.
The agent’s ability to solve the problem has not erased the fact that there was a problem.
Agent resilience and application resilience are different things
We want resilient testing agents. A brittle agent that gives up whenever something moves, takes slightly longer than expected, or follows an unexpected but valid path is not particularly useful. One of the reasons to use an agent in the first place is its ability to reason through that variability.
But we do not want that resilience to become a mask over application quality.
Agent resilience asks whether the testing system can adapt and continue. Application resilience asks whether the user can accomplish the task without needing unusual recovery behaviour.
A strong agent can have high resilience while testing an application with poor resilience. If we only measure whether the agent eventually completed the mission, those two signals collapse into one. And we lose information.
A test result doesn't have to be Boolean
Traditional automated testing has trained us to think heavily in terms of green and red. Pass. Fail.
Agentic testing opens up a much richer space because the execution itself contains signals. What surprised the agent? What did it retry? Where did it get stuck? What did it have to work around? Did it recover? How? Does the same recovery keep happening across multiple runs?
These aren't just incidental details about how the test ran. They're observations about the application.
Mission success ≠ application correctness
Test Result = Mission Outcome + Execution Evidence
The mission outcome tells us whether the objective was accomplished. The execution evidence tells us what had to happen to accomplish it. That second part becomes increasingly important as the testing agent gets smarter.
Let the agent be clever
The answer isn't to make agents deliberately stupid. I don't want an agent to fail immediately because something moved three pixels, a menu changed slightly, or an application took an unexpected but perfectly valid path.
I want it to reason. I want it to adapt. I want it to recover. The trick is making sure that recovery does not disappear from the result.
Making recovery visible in Testronaut
This isn't just a theoretical problem I've been thinking about. It's changed how Testronaut reports test results.
In the latest releases across the Testronaut, I've added recovery as a first-class mission outcome.
When an agent encounters a potential issue, finds a workaround, and still accomplishes its objective, the mission does not simply disappear into the pile of ordinary passes.
Instead, Testronaut can report: ⚠️ Passed after recovery.
The test still passes because the agent accomplished what it was asked to do. But the warning preserves an important second piece of information: the agent had to do something unusual to get there.
The recovery is also recorded as evidence. That gives the report a chance to show what the agent encountered, what recovery tactic it used, why it chose that tactic, and where it happened during the mission.
That's intentional. I don't want Testronaut deciding that every refresh, retry, detour, or unexpected login is definitely a bug. Sometimes it won't be. The agent may have encountered flaky infrastructure, a transient network problem, or some other harmless bit of web weirdness.
But I also don't want the agent silently fixing the situation and presenting a perfectly green report to the human reviewing it.
If two astronauts find an alien in the cargo bay, throw a tarp over it, and still get the shuttle home safely, I'm willing to call the mission a success.
I'd just prefer they mention the alien in the report...
...and then disect it in the medical bay later.
That is what the recovery status is meant to do. It lets the agent remain resilient without allowing that resilience to hide potentially meaningful defects.
We don't want to punish the agent for being clever, rather we want to make sure its cleverness leaves evidence behind.
Every time an agent has to be unusually clever, the report should tell us why.
Because as testing agents get better, I think the most interesting question isn't just: Did the test pass?
What uncouth MacGyver shenanigans did the agent do to make it pass?