--- title: 'AI in Software Testing: What Works | Testing Resources' description: A category-by-category look at where AI genuinely helps a test suite, where it does not, and how to evaluate a tool before you buy it. source_url: html: https://testingbot.com/resources/articles/ai-in-software-testing md: https://testingbot.com/resources/articles/ai-in-software-testing.md --- # AI in software testing: what actually works A category-by-category look at where AI genuinely helps a test suite, where it does not, and how to evaluate a tool before you buy it. By [Jochen D.](https://testingbot.com/about/jochen-d)2026-08-31 Share on Facebook Share on Twitter Post on Reddit Share link "AI in software testing" covers at least six different things, and they do not work equally well. Some are quietly useful and already in everyday use. One or two are close to solved. Others are demonstrated impressively and then fail on contact with a real codebase. This is a category-by-category assessment: what each thing actually does, how well it works today, and what it costs you when it goes wrong. It is written for someone deciding whether to put a tool in front of their suite, not for someone deciding whether AI is interesting. _Last reviewed 31 August 2026._ ## Table of Contents - [The six things people mean](https://testingbot.com#categories) - [1. Generating tests](https://testingbot.com#generation) - [2. Self-healing locators](https://testingbot.com#healing) - [3. Visual comparison](https://testingbot.com#visual) - [4. Agentic and autonomous exploration](https://testingbot.com#agentic) - [5. Failure triage](https://testingbot.com#triage) - [6. Test selection and flake detection](https://testingbot.com#selection) - [How to evaluate one honestly](https://testingbot.com#evaluate) - [What none of it fixes](https://testingbot.com#doesnt) ## The six things people mean The phrase is used for all of these, often in the same sentence, which is most of why the category is confusing: - **Generating tests** from a prompt, a user story or a recording - **Self-healing locators** that repair themselves when the markup changes - **Visual comparison** that decides whether a rendering difference matters - **Agentic exploration** , where a model drives the application and decides what to do next - **Failure triage** , turning a wall of red into a probable cause - **Test selection and flake detection** , deciding what to run and what to trust They have very different maturity. Roughly, the further down that list you go, the better it works, which is the opposite of the order the marketing tends to use. ## 1. Generating tests **What it does:** you describe a scenario, or point a tool at a page or a recording, and it writes a test. **How well it works:** genuinely well for the first draft, and this is the most useful thing on the list for most teams today. An LLM will produce a plausible [Playwright](https://testingbot.com/support/web-automate/playwright) or [Selenium](https://testingbot.com/support/web-automate/selenium) spec, with reasonable structure, far faster than you would type it. **Where it goes wrong:** the failure mode is not bad syntax, it is confident wrongness about your application. Generated tests routinely assert things that are true of a generic app and false of yours, invent selectors that do not exist, and pass for the wrong reason. A generated test that passes immediately deserves more suspicion than one that fails, because a test that cannot fail is worse than no test. **Practical guidance:** treat generated tests as a draft to review, exactly as you would a pull request from someone who has never seen the codebase. Make each one fail before you trust it. We cover the prompting workflow in detail in [test automation with ChatGPT](https://testingbot.com/resources/articles/test-automation-with-chatgpt). ## 2. Self-healing locators **What it does:** when a selector stops matching, the tool finds the element another way, using nearby text, position or other attributes, and carries on. Some tools also rewrite the selector for next time. **How well it works:** it does reduce the number of red builds caused by a class name changing. That is a real problem and this genuinely addresses it. **Where it goes wrong:** a test that repairs itself is a test that no longer tells you the UI changed. Sometimes the markup changed because someone refactored a stylesheet, and healing is correct. Sometimes it changed because the button now does something else, and healing hides a regression rather than finding one. The tool cannot tell those apart, because the difference is intent. **Practical guidance:** if you use it, treat every heal as an event to review rather than a success to celebrate. A tool that reports "healed 47 locators this week" is describing 47 unreviewed changes to what your tests mean. Also worth asking: healing is usually a workaround for brittle locators, and stable test ids fix the same problem permanently and for free. For the record, **TestingBot does not sell a self-healing feature.** This section is about the category, not about us. ## 3. Visual comparison **What it does:** compares screenshots between runs and decides whether a difference is a real regression or an acceptable variation such as antialiasing, a font fallback or an animation caught mid-frame. **How well it works:** this is the most solved thing on the list. Naive pixel diffing is famously noisy, and perceptual comparison is a genuine improvement rather than a repackaging. If you have ever abandoned visual testing because every run produced hundreds of false positives, this is the part of the category that fixed that. **Where it goes wrong:** tolerance is a judgement call, and a threshold loose enough to suppress noise is sometimes loose enough to suppress a real four-pixel misalignment. The other cost is baseline management, which is not an AI problem at all but is where most of the ongoing effort in visual testing actually goes. We cover the technique itself in [visual testing with Playwright](https://testingbot.com/resources/articles/playwright-visual-regression-testing) and [visual regression testing with Python](https://testingbot.com/resources/articles/python-visual-testing). ## 4. Agentic and autonomous exploration **What it does:** a model drives the application, decides what to try next, and reports what it found. This is the newest category and the one with the widest gap between demo and practice. **How well it works:** better than it did, and the reason is architectural rather than a bigger model. Agents that read a page's structure rather than looking at a picture of it are meaningfully more reliable, because they click a specific element rather than a guessed coordinate. Microsoft's Playwright MCP is the clearest example: it drives the browser through the accessibility tree, which it describes as needing "no vision models" and giving "deterministic tool application". We go through it in [Playwright MCP: what it does and where it fits](https://testingbot.com/resources/articles/playwright-mcp). **Where it goes wrong:** exploration produces findings, not tests. An agent that wanders your checkout and reports three oddities has done something useful, but you cannot put that in CI, because it does not do the same thing twice. There is also a real cost problem: an agent doing this continuously is expensive in a way a test suite is not. **Practical guidance:** the shape that works today is exploration as a source of test ideas, followed by a human deciding which ones are worth encoding as a deterministic test. "Autonomous testing" as a replacement for a maintained suite is not something we would advise planning around yet, and **TestingBot does not sell an autonomous testing product**. ## 5. Failure triage **What it does:** takes a failed run, its logs, its screenshots and its history, and proposes what went wrong. Groups failures with a common cause. **How well it works:** quietly one of the most valuable things on the list, and the least discussed. Summarising a large amount of correlated evidence into a probable cause is close to what language models are actually good at, and the cost of being wrong is low: a bad suggestion wastes a minute, whereas a bad self-heal hides a bug. **Where it goes wrong:** it is only as good as what it can see. Triage over a bare assertion message is guesswork. Triage over the full artefacts, the video, the console output, the network log and the previous ten runs of the same test, is a different and much better proposition. This is also where an agent connected to a grid earns its keep. The TestingBot MCP server exposes 48 tools that let an assistant pull builds, test details, run results and failure logs directly, so it can fetch the evidence rather than being handed a screenshot of it. Worth being precise, because the category invites overstatement: **it does not write or repair your tests.** There is no tool in it that generates or edits test code. See the [MCP documentation](https://testingbot.com/support/ai/mcp) and the [tool reference](https://testingbot.com/support/ai/mcp/tools). ## 6. Test selection and flake detection **What it does:** decides which tests to run for a given change, and identifies which of your tests are unreliable rather than genuinely failing. **How well it works:** the least glamorous category and arguably the highest return. Flake detection in particular is mostly statistics over run history rather than anything exotic, and it works because the signal is real: a test that passes on retry without a code change is telling you something measurable. **Where it goes wrong:** test selection is a risk trade, and the risk is silent. If the model declines to run the test that would have caught the bug, nothing tells you. Teams that adopt selection successfully generally keep a full run on a schedule as a backstop rather than trusting selection alone. We go into the flakiness problem itself in [how to fix flaky tests](https://testingbot.com/resources/articles/handling-test-flakiness). ## How to evaluate one honestly Most evaluations of this category go wrong the same way: the tool is trialled on a clean example application, performs well, and then behaves differently on the real codebase. A few things make a trial informative: - **Trial it on your worst area, not a clean one.** The legacy screen with the unlabelled divs is where the difference between tools shows up. A tool that does well on a tidy React app has told you very little. - **Measure the false-positive cost, not just the hit rate.**"Finds 90% of regressions" is meaningless without knowing how much noise came with it. A team that stops reading the output has gained nothing. - **Ask what happens when it is wrong.** A wrong triage suggestion costs a minute. A wrong self-heal costs you a missed regression. Weight the categories accordingly. - **Check whether it needs your source code or your data.** This is frequently the blocker, and it is better to find out in week one. - **Run it for long enough to see maintenance cost.** Nearly everything in this category demos well. The question is what it costs in month three. If you are comparing specific products rather than categories, our [AI testing tools roundup](https://testingbot.com/resources/articles/top-15-ai-testing-tools) is the page for that. ## What none of it fixes Three problems survive every tool in this category, and they are worth stating because they are usually the actual bottleneck: - **Knowing what to test.** Deciding which behaviours matter is a product judgement. A tool can generate a hundred tests for the wrong thing very efficiently. - **Environment and data.** A large share of "flaky" tests are neither flaky nor AI-addressable: they are a shared staging environment, a fixture that another test mutated, or a clock. No model fixes your test data strategy. - **Coverage across browsers and devices.** A test that is generated, self-healed and triaged brilliantly still only tells you about the one browser it ran in. That is a capacity question, not an intelligence one, and it is why a [real device and browser grid](https://testingbot.com/real-device-testing) sits underneath all of this rather than being replaced by it. If your suite is slow, the [parallel calculator](https://testingbot.com/support/parallel-calculator) shows what concurrency does to wall-clock time. The honest summary is that AI has meaningfully improved several parts of testing, mostly the unglamorous parts, and has not changed what a test suite is for. ## Related reading - [AI testing tools compared](https://testingbot.com/resources/articles/top-15-ai-testing-tools) - [Writing and interpreting tests with ChatGPT](https://testingbot.com/resources/articles/test-automation-with-chatgpt) - [Playwright MCP: what it does and where it fits](https://testingbot.com/resources/articles/playwright-mcp) - [Generative AI for Selenium test data](https://testingbot.com/resources/articles/generative-ai-selenium) - [How to fix flaky tests](https://testingbot.com/resources/articles/handling-test-flakiness) Topics [AI Testing](https://testingbot.com/resources/articles/topic/ai-testing) ## Sidebar ### TestingBot Cloud Testing Run automated, manual and visual tests on remote browsers and devices. Sign up for a free trial. [Free Trial](https://testingbot.com/users/sign_up) ### Latest articles [![blank](https://testingbot.com/assets/blank-765bda2d39d03b623c68515d78259503eb9ca884105d9fff58b2693418720ea1.gif)](https://testingbot.com/resources/articles/playwright-mcp) #### [Playwright MCP for Test Automation](https://testingbot.com/resources/articles/playwright-mcp) What Playwright MCP does, how driving the accessibility t... [Read article →](https://testingbot.com/resources/articles/playwright-mcp) [![15 Best AI Testing Tools in 2026](https://testingbot.com/assets/resources/articles/44-0566068e7f37e24257a058aa6ff3398f11e192e89be40e10d2e8a81335d16b6c.webp)](https://testingbot.com/resources/articles/top-15-ai-testing-tools) #### [15 Best AI Testing Tools in 2026](https://testingbot.com/resources/articles/top-15-ai-testing-tools) A reviewed list of 15 AI-driven testing tools for web and... [Read article →](https://testingbot.com/resources/articles/top-15-ai-testing-tools) [![Test Automation with ChatGPT](https://testingbot.com/assets/resources/articles/29-7c01b99e7284a560ad99094e3ce738c4942eee113a51295a0a8e9cd8fe0cbba5.webp)](https://testingbot.com/resources/articles/test-automation-with-chatgpt) #### [Test Automation with ChatGPT](https://testingbot.com/resources/articles/test-automation-with-chatgpt) How to prompt ChatGPT into generating Selenium, Appium, P... [Read article →](https://testingbot.com/resources/articles/test-automation-with-chatgpt) ## Other Articles [![Selenium and Generative AI](https://testingbot.com/assets/resources/articles/28-af707bf052844eb00f861bb6ebba57e6be4479dde1c78a5df4e70d2f597a8b18.webp)](https://testingbot.com/resources/articles/generative-ai-selenium "Selenium and Generative AI") ### [Selenium and Generative AI](https://testingbot.com/resources/articles/generative-ai-selenium) How to use generative AI to produce realistic test data for Selenium runs, and why varied data uncovers bugs that fixed fixtures never will. [Read article →](https://testingbot.com/resources/articles/generative-ai-selenium) ### [Parallel Testing: What Actually Scales](https://testingbot.com/resources/articles/parallel-testing) Why adding workers stops helping, how to shard so they are not idle, and what breaks the first time a suite runs concurrently. [Read article →](https://testingbot.com/resources/articles/parallel-testing) ### [Automated Regression Testing in CI](https://testingbot.com/resources/articles/automated-regression-testing) How to build a regression suite that stays fast and trustworthy in CI, what to leave out of it, and when to run which part. [Read article →](https://testingbot.com/resources/articles/automated-regression-testing) ### [Mobile App Test Automation at Scale](https://testingbot.com/resources/articles/mobile-app-test-automation) How to choose between Appium, Espresso, XCUITest and Maestro, and what changes when you run the suite on real devices at scale. [Read article →](https://testingbot.com/resources/articles/mobile-app-test-automation) ## Ready to start testing? [Start a free trial](https://testingbot.com/users/sign_up)