Don't Solve a $10 Testing Problem With $1,000 of Tokens

The easiest mistake to make with AI is using it to solve problems that are already cheap to solve.

Testing is no exception.

I've been thinking about this a lot while building Testronaut. Give the agent a goal, point it at an application, and it will navigate the UI, make decisions, recover from changes, and determine whether the original goal was accomplished.

Powerful stuff!

...But just because an agent can do something doesn't mean it should.

If I spin up a browser, send application context to an LLM, let an agent reason through several actions, capture screenshots, and generate a report just to determine whether an invalid email address produces an error message, I've probably solved a $10 problem with $1,000 worth of tokens over the lifetime of that piece of code (Ok I might be exaggerating, but you have to admit deep down this is true).

Agentic tests cost more to execute than unit tests. They're slower. They require a functioning environment. And every decision an agent makes potentially consumes tokens (yes, even wrong choices).

So where does paying that extra cost make sense?

The Testing Pyramid Still Matters

The traditional testing pyramid gives us a useful starting point.

At the bottom are unit tests. They're fast, cheap, deterministic, and narrow.

In the middle are integration tests, where we're interested in whether multiple pieces of the system work correctly together.

At the top are end-to-end tests, where we're testing the application much closer to the way an actual user experiences it.

As you move upward, the number of tests generally decreases while their scope increases.

There's a reason I like the pyramid metaphor here: you don't build a launchpad out of rockets.

The cheap, boring, repeatable tests at the bottom are what make the expensive tests at the top practical.

Agentic testing doesn't make this model obsolete.

If anything, I think it makes understanding the pyramid more important.

The question isn't:

Can an AI agent test this?

It almost certainly can.

The better question is:

Is this problem valuable enough to justify an AI agent testing it?

Start With the User Journey

I've found a simple heuristic useful when deciding where a test belongs:

Describe what you're testing without talking about the implementation.

If I say:

Given these inputs, calculateTotal() should return $53.42.

That's probably a unit test.

If I say:

When the API creates an order, the correct order and line items should be persisted to the database.

Now I'm talking about multiple pieces of the system interacting. That's probably an integration test.

But sometimes I find myself naturally describing a sequence:

A customer creates an account, verifies their email, signs in, selects a plan, completes checkout, and sees their subscription activated.

That's massively different.

I'm describing a user journey.

And that journey crosses a lot of boundaries:

UI → authentication → email → backend → database → payment provider → account state → UI

No unit/integration test can answer the question I actually care about here:

Can a customer successfully become a paying customer?

That's where moving higher up the pyramid starts to provide substantially more value. These are the core business promises and essential value propositions people practically care about.

When Does a Test Belong Higher Up the Pyramid?

There isn't a precise formula, but there are some useful signals.

A test becomes a stronger candidate for end-to-end testing when it:

• Requires multiple meaningful user actions
• Crosses multiple systems or application layers
• Relies on UI behaviour that materially affects the outcome
• Needs a fully functioning environment
• Means a workflow where failure has meaningful business impact

One of these items isn't necessarily enough and requires experienced judgement.

Testing to see that it correctly formats a phone number needs UI behaviour, but probably doesn't need an autonomous agent to verify it.

but...

Testing that clicking Upgrade, completing checkout, and returning to the application correctly updates a customer's permissions is a much better candidate.

The important distinction is the risk being covered.

Think About Risk, Not Just Coverage

We often talk about testing in terms of code coverage, but when deciding whether an expensive test is worthwhile, I think risk coverage is a more useful concept.

Imagine the two tests above again:

The first verifies that a utility function correctly formats a phone number...

and the second verifies that customers can complete checkout.

Both of these do matter, but if the first breaks, the consequences might be an incorrectly formatted phone number.

If the second breaks, the company might stop making money! (and you'll need to update your resume)

The first test isn't unnecessary, but it does mean the cheapest appropriate testing mechanism should probably cover it.

A useful way to think about it is:

Test ROI = Risk Covered ÷ Total Cost of Testing

And, for the security-minded readers who will find this familiar, risk itself can roughly be thought about as:

Likelihood of Failure × Impact of Failure × Confidence Provided by the Test

These aren't numbers I expect anyone to put into a spreadsheet. Instead, they're a way of thinking about what you're buying with each test.

The Economics Change With Agents

There's another important part of this equation. Execution isn't the only cost of testing.

A better representation of total cost might be:

Authoring + Maintenance + Execution + Investigation

Agentic end-to-end testing gets interesting here because traditional end-to-end automation often encodes the mechanics of a workflow:

Find this selector. Click this button. Wait for this element. Fill this field. Find the next selector.

But those mechanics aren't really what we care about.

The actual requirement might simply be:

Buy the product.

When the UI changes, a traditional test may fail even though a customer could still successfully accomplish the goal, which creates maintenance work.

Alternatively, an agent can instead reason about the interface in front of it and pursue the goal. A button can move. A selector can change. An extra confirmation screen can appear.

If the path changes but the goal remains achievable, the test may still succeed.

As a result, agentic testing changes the math where you're trading higher execution cost for lower authoring and maintenance cost, while gaining the ability to test complex workflows in terms closer to actual user intent.

That trade is worth making in some places most definitely...

...but not everywhere.

Spend Expensive Tests on Expensive Questions

This is ultimately how I think about agentic testing:

Unit tests are incredibly cheap and precise. Their value comes in volume (We can have thousands of small assertions protecting thousands of small behaviours).

Integration tests cost a little more, but give us confidence that important pieces of our architecture actually work together.

End-to-end tests are more expensive again.

Agentic end-to-end tests add another cost: reasoning.

And to me that means they should earn their place.

Don't spend tokens asking an autonomous agent whether a function returns the right value, and don't launch a browser with an LLM just to prove that a required field displays an error message.

Spend those tokens answering expensive questions:

• Can a new customer create an account and get started?
• Can an existing customer upgrade their subscription?
• Can an administrator invite an employee and assign the correct permissions?
• Can a user recover their account when they've forgotten their password?
• Can a customer actually buy the thing we're selling?

Those aren't tests of individual pieces of code, but rather tests of whether the business and product works.

Agentic testing doesn't mean we should turn the testing pyramid upside down. Instead, it means we have a new tool available near the top of it.

And if one of your most important user journeys unexpectedly starts answering "no," a few dollars of tokens will probably be the cheapest part of your day.