Our AI wrote the same bug twice.

Three weeks apart. Different screen. Nobody edited anything in between.

We found it the first time, fixed it, moved on. Then the next module came back with the same broken login check. The one that thinks you are signed in while the browser is still sitting on the login page.

That is not a model problem.

You can change the model and it happens again. It happens because there is nowhere in our system for a fact to live.

So that is what we are building. We call it the Application Profile.

The idea is small. Every time the system learns something true about a customer's application, it gets written down once and reused.

• The route to a page
• The exact wording of a success message
• Whether a field is required
• What the app does when you leave it blank
• What has to finish loading before the next step

Each fact carries a status. Proven. Unproven. Out of date.

When the application changes, the fact expires and has to be established again. It does not quietly go stale and take a hundred tests down with it.

Two things pushed us here faster than the bug did.

The first is cost. Almost everything we spend on a run goes on re-reading the application, not on writing tests. Every module starts by working the same things out again. There is no second-time discount. The tenth module costs what the first one did.

The second surprised me.

In our last run, the cheapest model started doing this on its own. Instead of guessing what a field allows, its tests said where it had read the rule and named the source file.

Nobody asked it to. It stopped guessing because it had somewhere better to look.

One thing I want to be clear about, because it is the part people get wrong.

This is not caching.

A cached guess is still a guess. A fact only counts once something has run and shown it to be true, and a person has signed off. Skip that and you have only made the system confidently wrong, faster.

We are building it now. I will report back on what breaks.

If you are generating tests with AI, where does what it learns about your application actually live?

Last week I wrote that the AI model which passed every test actually produced the worst test suite.

This week we found out why the best-written suite failed completely.

One wrong string.

The tests checked whether login worked by looking for "signin" in the URL.

The application uses "sign-in".

Because of that small difference, the login check gave a false positive. The browser was still sitting on the login page, but the tests continued.

All five tests then failed about twenty seconds later, for reasons that had nothing to do with the real problem.

Here's the part that bothered me most.

We had already found this exact mistake earlier in the same project, on a different screen of the same application.

We fixed it.

Then, when we generated tests again, the AI made the same mistake.

That made something very clear:

The model wasn't the main problem. Our architecture was.

Every generation started from zero.

So the AI had to rediscover the application every time:

• Where is the login page?
• How do I know login actually succeeded?
• What is the upload field called?
• What success message should appear?
• What does the application need to finish loading before the next step?

If the model doesn't know something, it guesses.

And a guess that looks reasonable is often the most expensive kind of wrong.

So we're changing the architecture around three ideas:

1. Remember what the system learns.

Store verified facts about each application once and reuse them instead of asking the AI to rediscover them.

2. Stop regenerating shared setup code.

Write login, navigation and common setup once, review it carefully, and have every generated test reuse it.

3. Make sure every test can actually fail.

Before a person reviews a generated test, deliberately break the condition it is supposed to check and run it again. If the test still passes, it wasn't testing anything meaningful.

None of this is built yet.

We start this week, and I'll share what actually happens - including the parts that don't work.

If your team is using AI to generate tests:

Where do you store what the AI has already learned about your application?

We pointed our AI accelerator at the same application three times, changed nothing but the model doing the work, and got three very different answers.

Same app. Same instructions. All three found the same modules. After that they diverged completely.

The first model wrote the most thorough single test of the three. Only one test file, but it covered the edge cases and the security-adjacent paths, and its filter test exercised every filter on the screen rather than one or two. Its weakness was how it waited: fixed pauses, hoping the screen had caught up. One test timed out on a modal button, and when the framework tried to repair it, it couldn't. It produced the least output for the most money.

The second spent the least money of the three, and everything it wrote passed on the first run. That sounds like the winner until you look at what it actually wrote. Positive and negative cases only, no boundary or security testing, basic navigation waits, and assertions that mostly checked whether text had appeared on screen. It also reported by far the most test flows, which looked like better discovery until we read the list and found the same flow three times under different names. A clean pass rate on a shallow suite isn't a good result. It's the easiest result to mistake for one.

The third produced the most usable work by some distance. Several test files, full scenario coverage, and the only one that waited properly — it kept checking until the application actually reached the expected state, and let the loading placeholders settle before asserting anything. It was also the only one to check the network calls behind the page, test a real file download, and verify the file landed on disk. It failed on its first run, repaired itself, and finished clean.

Here is the part worth sitting with. It used far more tokens than the other two and its total bill was several times higher than the cheapest model's — and it still came out the lowest cost per test file of the three, because it produced so much more usable work. The smallest invoice bought the least. The larger invoice bought the most per dollar spent.

Three things I didn't expect to be this stark.

Pass rate is a trap. The model that passed everything wrote the least worth passing.

The quality difference lives in how tests wait, not in what they check. Fixed waits are why teams end up muting their own suites six months later, and that never shows up in a coverage number.

More discovered flows isn't more coverage. Counting the same flow three times is just maintenance you'll pay for later. We now collapse duplicates before anything becomes a test.

This is most of why we run our AI Accelerator for QA as a service rather than selling it as a tool. We pick the model and re-test as models change, we collapse duplicate flows, and someone here signs off on every generated test before it reaches your repo.

If your team is already generating tests with AI: how did you choose the model, and when did you last check that choice? I'd like to hear what comes back — and if you'd rather not answer that publicly, a DM is fine.

Playwright's become the default choice for a lot of engineering teams building out browser test coverage — it's fast, it plays well with modern CI/CD, and downloads have pulled well past where Cypress and Selenium sit today.

That shift has made it easier than ever to generate Playwright tests with AI. Which sounds like exactly what QA teams have been asking for, until you notice what it actually produces: a lot of green checkmarks, and no easy way to tell whether they're testing anything that matters.

AI is probabilistic. Testing needs to be deterministic. A generated test can pass without validating the business logic it was supposed to cover — and once a team is generating hundreds of tests a week, nobody's reading each one closely enough to catch that.

We built our AI Accelerator for QA to plug directly into an existing Playwright suite. Every test case it produces still goes through an approval gate your team controls, because green tests aren't the same thing as safe tests.

If your team's already leaning on AI to write Playwright tests, I'm curious how you're keeping tabs on what's actually getting validated — comment or send me a DM, happy to compare notes.