From Test Plan to Playwright Test with an Agent

The most dependable use of an AI agent in testing today is turning a written scenario into Playwright code that you then review like any other pull request. This chapter covers the input that gets good output, a real batch of generated tests with the problems they had, and the checklist that decides whether generated code is merged.

Set up the project#

Start from a normal Playwright project (the first chapter of the Playwright course walks through npm init playwright@latest). The config used in this course is deliberately small:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  reporter: [['list'], ['html', { open: 'never' }]],
  use: {
    baseURL: 'https://demo.playwright.dev',
    trace: 'on-first-retry',
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
});

baseURL matters more with agents than without: if the agent can write page.goto('/todomvc/'), it cannot hard-code a staging hostname you later have to hunt down.

Write the plan, not the prompt#

The agent needs to know what should happen, not just what to click. A useful scenario has a starting state, steps, and an expected result for each step, written as requirements:

#### Complete a single todo

Starting state: empty list.

1. Add "Buy groceries", "Walk the dog" and "Read a book".
   - expect: three unchecked todos, counter says "3 items left".
2. Tick "Walk the dog".
   - expect: it is marked completed, the other two are not,
     counter says "2 items left", a "Clear completed" button appears.
3. Untick "Walk the dog".
   - expect: counter back to "3 items left", "Clear completed" disappears.

Then the prompt to any MCP-capable agent with Playwright MCP enabled:

Write a Playwright Test file in tests/todomvc/complete-todo.spec.ts for the scenario in specs/todomvc.md, section "Complete a single todo". Execute each step in the browser first to find the elements. Use getByRole, getByLabel or getByTestId locators, web-first assertions (await expect(...)), and no waitForTimeout. Assert every "expect" line from the plan; do not add assertions the plan does not ask for. Do not run or modify any other file.

Each sentence there exists because of a failure we have seen. "Execute each step first" stops it inventing selectors from what TodoMVC usually looks like. "Do not add assertions the plan does not ask for" stops it from asserting incidental details such as CSS. "Do not modify any other file" keeps the diff reviewable.

If you would rather not write plans by hand, chapter 3 has an agent draft them. Treat a drafted plan as something you edit and approve, because every expected result in it becomes an assertion.

What came back#

The four TodoMVC tests in this course were generated by the Playwright generator agent from chapter 3, working from a plan like the one above. The review below applies whichever tool wrote the code. Three tests passed on the first run. The fourth did not:

Error: expect(locator).not.toBeChecked() failed

Locator: getByLabel('Toggle Todo')
Expected: not checked
Error: strict mode violation: getByLabel('Toggle Todo') resolved to 3 elements

The generated line was:

await expect(page.getByLabel('Toggle Todo')).not.toBeChecked();

The intent ("none of the three todos is checked") is right, but toBeChecked asserts on exactly one element, and Playwright's strict mode refuses to guess which of three you meant. This is the most common class of mistake in generated tests: code that reads correctly and expresses the right idea, but does not do what the API actually does.

Other things the review found in the same batch:

• An implementation-detail assertion. The plan said "marked completed"; the agent also asserted toHaveCSS('text-decoration-line', 'line-through'). It passes today and breaks when a designer changes the strikethrough to greyed-out text, a change no user would call a bug. We removed it and kept the completed class check, which reflects the app's own state.
• An unverified expectation that happened to be true. The plan, drafted by an agent, said leading and trailing spaces are trimmed from new todos. The planner never typed spaces into the app; it wrote that from its general knowledge of TodoMVC. The test passes because this demo does trim. On your app that line would be a guess, written down as if it were a requirement.
• A wrong reference. Each file starts with // spec: specs/plan.md, while the plan was in specs/todomvc.md. Harmless, but it shows that generated metadata is not checked by anything.

The fix we merged#

The healer agent (chapter 3) fixed the strict mode error by looping:

for (const checkbox of await page.getByLabel('Toggle Todo').all()) {
  await expect(checkbox).not.toBeChecked();
}

That passes, but .all() takes a snapshot of the matching elements at that moment and does not wait for the list to settle. In review we replaced it with one retrying assertion that says the same thing:

await expect(page.getByRole('checkbox', { name: 'Toggle Todo', checked: true })).toHaveCount(0);

The point is not that the agent's version was wrong. It was acceptable code that a reviewer who knows Playwright would write differently. Generated code drifts toward "passes" rather than "idiomatic", and reviews pull it back.

The acceptance checklist#

Run through this for every generated test before it is merged. It is short on purpose, so people actually use it.

1. Every assertion traces back to a requirement. For each expect, can you point to the plan line, ticket or spec that says this is correct? If the only source is "the page showed it", either find the requirement or delete the assertion.
2. Locators describe what users see. getByRole, getByLabel, getByPlaceholder, getByText, or getByTestId when there is no stable accessible name. No .nth(), .first() or positional XPath unless order is the thing being tested.
3. Web-first assertions only. await expect(locator).toBeVisible(), not expect(await locator.isVisible()).toBe(true), which checks once and never retries.
4. No fixed waits. No waitForTimeout, no waitForLoadState('networkidle'). The instructions Playwright ships for its healer agent forbid networkidle, and so should your review.
5. No implementation details (CSS values, internal class names that are not state, timing) unless that detail is the requirement.
6. Independent tests. Each test sets up its own data and passes alone and in any order.
7. No secrets or personal data in the code, the comments or the test names.
8. Proven stable. Run the new tests several times before merging.

Items 2 to 4 can be checked mechanically. A simple search catches most violations:

grep -rnE "waitForTimeout|networkidle|\.nth\(|\.first\(\)|\.last\(\)" tests/

It returned nothing on our reviewed suite. For a stronger check, the community eslint-plugin-playwright package has rules for most of these patterns; we did not run it for this course. Item 8 is one command:

npx playwright test tests/todomvc --repeat-each=5

Our four reviewed tests ran 20 times and all 20 passed. A generated test that fails once in twenty runs locally will fail more often in CI.

Next, the tool that wrote these tests: the Playwright Test Agents, run end to end, with their costs and their failure modes.