By Jochen D.
Most regression suites do not fail because the tests were badly written. They fail because the suite grew faster than anyone's willingness to maintain it, until it took too long to run on every change and failed often enough that people stopped reading the results.
This is about avoiding that. What belongs in a regression suite, what does not, when to run which part, and what to do when it gets slow or noisy.
Last reviewed 31 August 2026.
What automated regression testing is for
A regression suite answers one question: did this change break something that used to work? Not whether the new feature is any good, not whether the design is right. Only whether the things that worked yesterday still work.
That narrowness is the point, and it is what makes automation worthwhile here specifically. Checking the same journeys after every change is repetitive, unrewarding and exactly the sort of work people quietly skip under deadline pressure. It is also the work a machine does perfectly.
If you want the wider definitions and the taxonomy around it, our what is regression testing page covers that ground. This page assumes you have decided to automate and are trying to make it survive contact with a real CI pipeline.
Why suites rot
The failure mode is nearly always the same sequence, and it is worth recognising early because each step looks reasonable on its own:
- The suite grows, because adding a test is easy and deleting one feels risky.
- It gets slower, so it moves from every commit to nightly.
- Feedback now arrives a day after the change that caused it, so failures are harder to attribute.
- A few tests become unreliable. Reruns become normal.
- People start assuming red means flaky, and a real regression ships.
Notice that the technical problem, slowness, becomes a trust problem, and the trust problem is the one that actually costs you. A slow suite is annoying. A suite nobody believes is worse than no suite at all, because it costs money and provides no signal.
What to put in the suite
The instinct is to automate everything that was ever manually tested. That is how you get the rot above. A more useful filter is to ask what you would actually stop a release for.
Strong candidates: the journeys that make money or lose it. Sign-up, log-in, checkout, payment, whatever your equivalent of adding to a basket is. Anything that has broken before, because bugs cluster. Anything with a long, quiet blast radius, such as billing.
Poor candidates: anything still changing shape weekly, which will cost more in maintenance than it returns. Detailed permutations that a unit test can cover far faster and closer to the code. Cosmetic checks that a human notices instantly and an assertion describes badly. Anything you would not actually block a release for, which is a surprisingly large share of most suites.
A useful discipline: every test in the suite should have an answer to "what would we do if this failed at 5pm on a Friday?". If the answer is "ignore it until Monday", it does not belong in the suite that gates a release. Move it or delete it.
When to run which part
Trying to run everything on every commit is what forces the move to nightly, and nightly is where attribution dies. Splitting by when you need the answer works better than splitting by feature area:
- On every commit: a small, fast, ruthlessly stable set. Minutes, not tens of minutes. If it is red, the build is broken and everybody believes it.
- On every pull request: the core journeys, across the browsers most of your users actually use. This is the tier that catches genuine regressions before review.
- Nightly or pre-release: the long tail. Older browser versions, edge-case data, the combinations that matter but not urgently.
The first tier is the one people get wrong, usually by making it too big. Its job is not coverage. Its job is to be trusted absolutely, so that a red build stops work without argument. A single flaky test in that tier undoes it.
Keeping it trustworthy
Trust is the resource this whole exercise spends, so it is worth protecting deliberately:
- Treat a flaky test as a broken test. A test that passes on retry has told you something is wrong, either in the test or in the application. Quarantine it out of the gating tier, but track it; a quarantine that only ever accumulates is a way of hiding the problem rather than fixing it.
- Never fix a failure by widening a wait. Waiting for a condition is correct. Waiting for a duration is a guess that will be wrong on a slower machine, and CI machines are slower than laptops.
- Make tests independent. Tests that must run in order, or that depend on data another test created, fail in ways that look random the moment you parallelise them.
- Assert on the thing you care about. A test that fails because a heading changed, when it was meant to check a payment succeeded, trains people to ignore it.
The mobile and browser-specific causes of flakiness are worth knowing separately; we go through them in how to fix flaky tests.
Keeping it fast
There are only really three levers, and they are not equally good:
- Run fewer tests. The tiering above, plus actually deleting tests that no longer earn their place. Underused and the most effective.
- Run them at the same time. The main lever once the suite is genuinely necessary, and the reason wall-clock time and total run time stop being the same number. It has sharper limits than people expect, since setup time and your slowest single test both cap the gain no matter how many workers you add; parallel testing: what actually scales goes through the arithmetic, and the parallel calculator does it for your numbers.
- Make each test faster. Real but limited. Seeding state through an API rather than clicking through the UI to reach it is usually the biggest single win, and it also removes a class of flakiness.
A suite that is parallel-safe is a precondition for the second lever, which is another reason the independence rule above matters more than it looks.
Running it across browsers
A regression suite that only runs in one browser answers a narrower question than people assume. It tells you the logic still works. It does not tell you the thing still works for the person on the browser you do not develop in.
The practical approach is to match browser coverage to the tier: the commit tier in one browser for speed, the pull-request tier across the handful your analytics say matter, and the full matrix nightly. Running those in parallel on a grid is what keeps the second tier inside a reasonable pull-request wait, and TestingBot provides the browsers and real devices so that the matrix is a configuration choice rather than a hardware purchase.
Whichever framework you write in, the suite itself does not change: Selenium, Playwright and Cypress all point at a remote grid with a configuration change rather than a rewrite.
Knowing whether it is working
Pass rate is a poor measure, because a suite that always passes might be catching nothing. More useful signals:
- Escaped defects. Bugs found in production that the suite should have caught. The only measure that directly answers whether the suite is doing its job.
- Time from push to feedback for each tier, which is what determines whether people act on the result.
- Retry rate. Rising retries are trust draining away, usually before anyone says so out loud.
- Tests deleted. A suite that only ever grows is not being maintained, it is being accumulated.
If you track one thing, track escaped defects against retry rate. A suite catching real problems with a low retry rate is working. A suite with a high retry rate and no escaped defects is probably not being read.