--- title: From Test Plan to Playwright Test with an Agent | TestingBot description: The most dependable use of an AI agent in testing today is turning a written scenario into Playwright code that you then review like any other pull request.… source_url: html: https://testingbot.com/resources/courses/ai-test-automation/generate-tests-from-a-plan md: https://testingbot.com/resources/courses/ai-test-automation/generate-tests-from-a-plan.md --- Course [AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) Chapter 3 of 9 1. [1 AI Test Automation: A Hands-On Course](https://testingbot.com/resources/courses/ai-test-automation) 2. [2 Set Up an Agent with Playwright MCP](https://testingbot.com/resources/courses/ai-test-automation/set-up-playwright-mcp) 3. [3 From Test Plan to Playwright Test with an Agent](https://testingbot.com/resources/courses/ai-test-automation/generate-tests-from-a-plan) 4. [4 Playwright Test Agents: Planner, Generator and Healer](https://testingbot.com/resources/courses/ai-test-automation/playwright-test-agents) 5. [5 Self-Healing Tests: When Healing Hides a Regression](https://testingbot.com/resources/courses/ai-test-automation/self-healing-tests) 6. [6 AI for Failure Triage: Traces, Logs and Privacy](https://testingbot.com/resources/courses/ai-test-automation/ai-failure-triage) 7. [7 Run AI-Written Suites on TestingBot](https://testingbot.com/resources/courses/ai-test-automation/run-on-testingbot) 8. [8 Guardrails for AI in a Test Suite](https://testingbot.com/resources/courses/ai-test-automation/guardrails) 9. [9 AI Test Automation Quiz](https://testingbot.com/resources/courses/ai-test-automation/knowledge-check) [Exit course](https://testingbot.com/resources/courses) Chapter 3 of 9 Intermediate 8 min read 2 quick checks Updated Nov 2026 # From Test Plan to Playwright Test with an Agent On this page 1. [Set up the project](https://testingbot.com#set-up-the-project) 2. [Write the plan, not the prompt](https://testingbot.com#write-the-plan-not-the-prompt) 3. [What came back](https://testingbot.com#what-came-back) 4. [The fix we merged](https://testingbot.com#the-fix-we-merged) 5. [The acceptance checklist](https://testingbot.com#the-acceptance-checklist) The most dependable use of an AI agent in testing today is turning a written scenario into Playwright code that you then review like any other pull request. This chapter covers the input that gets good output, a real batch of generated tests with the problems they had, and the checklist that decides whether generated code is merged. ## Set up the project[#](https://testingbot.com#set-up-the-project) Start from a normal Playwright project (the [first chapter of the Playwright course](https://testingbot.com/resources/courses/playwright/install-and-first-test) walks through `npm init playwright@latest`). The config used in this course is deliberately small: ```typescript import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ testDir: './tests', fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, reporter: [['list'], ['html', { open: 'never' }]], use: { baseURL: 'https://demo.playwright.dev', trace: 'on-first-retry', }, projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }], }); ``` `baseURL` matters more with agents than without: if the agent can write `page.goto('/todomvc/')`, it cannot hard-code a staging hostname you later have to hunt down. ## Write the plan, not the prompt[#](https://testingbot.com#write-the-plan-not-the-prompt) The agent needs to know what _should_ happen, not just what to click. A useful scenario has a starting state, steps, and an expected result for each step, written as requirements: ```markdown #### Complete a single todo Starting state: empty list. 1. Add "Buy groceries", "Walk the dog" and "Read a book". - expect: three unchecked todos, counter says "3 items left". 2. Tick "Walk the dog". - expect: it is marked completed, the other two are not, counter says "2 items left", a "Clear completed" button appears. 3. Untick "Walk the dog". - expect: counter back to "3 items left", "Clear completed" disappears. ``` Then the prompt to any MCP-capable agent with Playwright MCP enabled: > Write a Playwright Test file in tests/todomvc/complete-todo.spec.ts for the scenario in specs/todomvc.md, section "Complete a single todo". Execute each step in the browser first to find the elements. Use getByRole, getByLabel or getByTestId locators, web-first assertions (await expect(...)), and no waitForTimeout. Assert every "expect" line from the plan; do not add assertions the plan does not ask for. Do not run or modify any other file. Each sentence there exists because of a failure we have seen. "Execute each step first" stops it inventing selectors from what TodoMVC usually looks like. "Do not add assertions the plan does not ask for" stops it from asserting incidental details such as CSS. "Do not modify any other file" keeps the diff reviewable. If you would rather not write plans by hand, chapter 3 has an agent draft them. Treat a drafted plan as something you edit and approve, because every expected result in it becomes an assertion. ## What came back[#](https://testingbot.com#what-came-back) The four TodoMVC tests in this course were generated by the Playwright generator agent from chapter 3, working from a plan like the one above. The review below applies whichever tool wrote the code. Three tests passed on the first run. The fourth did not: ``` Error: expect(locator).not.toBeChecked() failed Locator: getByLabel('Toggle Todo') Expected: not checked Error: strict mode violation: getByLabel('Toggle Todo') resolved to 3 elements ``` The generated line was: ```typescript await expect(page.getByLabel('Toggle Todo')).not.toBeChecked(); ``` The intent ("none of the three todos is checked") is right, but `toBeChecked` asserts on exactly one element, and Playwright's strict mode refuses to guess which of three you meant. This is the most common class of mistake in generated tests: code that reads correctly and expresses the right idea, but does not do what the API actually does. Other things the review found in the same batch: - **An implementation-detail assertion.** The plan said "marked completed"; the agent also asserted `toHaveCSS('text-decoration-line', 'line-through')`. It passes today and breaks when a designer changes the strikethrough to greyed-out text, a change no user would call a bug. We removed it and kept the `completed` class check, which reflects the app's own state. - **An unverified expectation that happened to be true.** The plan, drafted by an agent, said leading and trailing spaces are trimmed from new todos. The planner never typed spaces into the app; it wrote that from its general knowledge of TodoMVC. The test passes because this demo does trim. On your app that line would be a guess, written down as if it were a requirement. - **A wrong reference.** Each file starts with `// spec: specs/plan.md`, while the plan was in `specs/todomvc.md`. Harmless, but it shows that generated metadata is not checked by anything. A generated test contains `await expect(page.getByRole('row')).toContainText('Paid')` and fails with a strict mode violation on a table with 12 rows. What is the right fix? Add `.first()`, since the first row is the one the agent created Increase the assertion timeout Switch to a CSS selector such as `tr:nth-child(1)` Filter to the row the test cares about, for example `getByRole('row').filter({ hasText: 'INV-1042' })`, then assert on it Strict mode is pointing out that the intent is ambiguous. Filtering by the data the test created states the intent. `.first()` and `nth-child` depend on row order, which is exactly the hidden assumption strictness exists to expose. ## The fix we merged[#](https://testingbot.com#the-fix-we-merged) The healer agent (chapter 3) fixed the strict mode error by looping: ```typescript for (const checkbox of await page.getByLabel('Toggle Todo').all()) { await expect(checkbox).not.toBeChecked(); } ``` That passes, but `.all()` takes a snapshot of the matching elements at that moment and does not wait for the list to settle. In review we replaced it with one retrying assertion that says the same thing: ```typescript await expect(page.getByRole('checkbox', { name: 'Toggle Todo', checked: true })).toHaveCount(0); ``` The point is not that the agent's version was wrong. It was acceptable code that a reviewer who knows Playwright would write differently. Generated code drifts toward "passes" rather than "idiomatic", and reviews pull it back. ## The acceptance checklist[#](https://testingbot.com#the-acceptance-checklist) Run through this for every generated test before it is merged. It is short on purpose, so people actually use it. 1. **Every assertion traces back to a requirement.** For each `expect`, can you point to the plan line, ticket or spec that says this is correct? If the only source is "the page showed it", either find the requirement or delete the assertion. 2. **Locators describe what users see.** `getByRole`, `getByLabel`, `getByPlaceholder`, `getByText`, or `getByTestId` when there is no stable accessible name. No `.nth()`, `.first()` or positional XPath unless order is the thing being tested. 3. **Web-first assertions only.** `await expect(locator).toBeVisible()`, not `expect(await locator.isVisible()).toBe(true)`, which checks once and never retries. 4. **No fixed waits.** No `waitForTimeout`, no `waitForLoadState('networkidle')`. The instructions Playwright ships for its healer agent forbid `networkidle`, and so should your review. 5. **No implementation details** (CSS values, internal class names that are not state, timing) unless that detail is the requirement. 6. **Independent tests.** Each test sets up its own data and passes alone and in any order. 7. **No secrets or personal data** in the code, the comments or the test names. 8. **Proven stable.** Run the new tests several times before merging. Items 2 to 4 can be checked mechanically. A simple search catches most violations: ```bash grep -rnE "waitForTimeout|networkidle|\.nth\(|\.first\(\)|\.last\(\)" tests/ ``` It returned nothing on our reviewed suite. For a stronger check, the community `eslint-plugin-playwright` package has rules for most of these patterns; we did not run it for this course. Item 8 is one command: ```bash npx playwright test tests/todomvc --repeat-each=5 ``` Our four reviewed tests ran 20 times and all 20 passed. A generated test that fails once in twenty runs locally will fail more often in CI. A generated test passes, follows every locator and waiting rule, and asserts that the order confirmation shows "Delivery in 5-7 days". Nobody can find where that delivery estimate is specified. What should happen? Find the requirement or remove the assertion; a passing test does not make the value correct Merge it, since the test is stable and the page showed that text Replace it with a regular expression so it accepts any delivery estimate Checklist item 1: an assertion is a claim about correct behaviour. If the only source is the current screen, the test turns today's behaviour, bugs included, into the specification. Loosening it to a regex hides the question rather than answering it. Next, the tool that wrote these tests: the Playwright Test Agents, run end to end, with their costs and their failure modes. [Previous · Chapter 2 Set Up an Agent with Playwright MCP](https://testingbot.com/resources/courses/ai-test-automation/set-up-playwright-mcp) [Next · Chapter 4 Playwright Test Agents: Planner, Generator and Healer](https://testingbot.com/resources/courses/ai-test-automation/playwright-test-agents) On this page 1. [Set up the project](https://testingbot.com#set-up-the-project) 2. [Write the plan, not the prompt](https://testingbot.com#write-the-plan-not-the-prompt) 3. [What came back](https://testingbot.com#what-came-back) 4. [The fix we merged](https://testingbot.com#the-fix-we-merged) 5. [The acceptance checklist](https://testingbot.com#the-acceptance-checklist) Certification This course has a free, hands-on certificate. [How it works](https://testingbot.com/resources/courses/ai-test-automation/certification)