Guardrails for AI in a Test Suite

The earlier chapters each ended with a rule of thumb. This one collects them into a policy you can write into a repository and enforce in CI. None of it is specific to one AI vendor, and most of it is ordinary engineering practice applied to a new kind of contributor: one that works fast, never gets tired, and has no idea what your product is for.

1. AI writes code at authoring time; CI runs code#

The most important decision is where the model runs. There are two options:

• At authoring time: an agent writes or edits test files, a person reviews the diff, and CI runs the committed code. Every CI run does the same thing for the same commit, and a failure points at code you can read.
• At run time: a model interprets a prompt, picks elements, or repairs locators while the suite runs in CI. The same commit can behave differently on two runs, a model or provider change alters results without a commit, and every run costs tokens.

Keep the model at authoring time for the suite that gates releases. Run-time AI has its place in exploratory checks, monitoring and smoke tests that tolerate some noise, but not in the build that decides whether you ship. If a tool you evaluate heals or interprets at run time, ask how you review what it decided and how you reproduce last Tuesday's run.

2. Prompts and agent definitions live in the repository#

Prompts shape your tests as much as code does. Commit them:

• the agent definitions from npx playwright init-agents (.claude/agents/, .github/agents/ or .codex/agents/), and the MCP config next to them;
• the test plans in specs/, which are the oracle for generated tests;
• the prompt templates your team reuses, such as the generation prompt from chapter 2 and the triage prompt from chapter 5.

Then changes to them go through review, and when generated tests suddenly start to look different, git log on the agent definitions tells you why. Pin MCP server versions in those configs (@playwright/mcp@0.0.85, not @latest), and re-run init-agents deliberately when you upgrade Playwright, reviewing the diff of the definitions like a dependency update.

3. Secrets stay out of prompts, plans and code#

An agent reads everything in its workspace and sends what it reads to a model provider. Assume anything the agent can see may leave your network.

• Credentials come from environment variables or your CI secret store. TB_KEY and TB_SECRET in this course never appear in a file.
• .env files, playwright/.auth/ storage state and .playwright-mcp/ output are in .gitignore, and you keep them out of directories you point an agent at, or deny the agent read access to them in your client's permission settings.
• Test accounts are test accounts: no production admin logins, no copies of real customer data in fixtures.
• Playwright MCP's --secrets file masks known values in tool output. It is a useful extra layer, and the README calls it a convenience, not a security feature.
• Never paste a trace.zip, a HAR file or a log into a chat service your company has not approved for that data (chapter 5).

There is also a newer risk: prompt injection. An agent that browses reads page content, and page content can contain instructions ("ignore previous instructions and..."). On your own staging app that is unlikely; on third-party pages, user-generated content or anything an attacker can write to, it is real. The Playwright MCP README says plainly that the server is not a security boundary. Limit what the agent can do (no shell access it does not need, no production credentials, --allowed-origins to keep it on your sites) rather than relying on the model to resist.

4. Budgets for tokens and grid minutes#

From this course's own runs, at list price as reported by Claude Code with a Sonnet model:

These are small numbers, and they are per run. A healer wired to run on every failed CI job, on a suite with a hundred flaky tests, runs many times a day. Agents also spend more when they get stuck: in chapter 6, an agent that could not find a build by name kept paging through results. Set a spending limit in your AI provider account, prefer one-off runs started by a person over automatic triggers, and log what each automated run cost.

Grid minutes are the other meter. Tests run by an agent through the TestingBot MCP tb_* tools are real sessions billed like any other, and an agent that forgets to close a browser keeps a session open until the idle timeout. Ask agents to close what they open.

5. A CI policy for AI contributions#

Write it down, put it in the repository (for example in CONTRIBUTING.md), and enforce the parts a machine can check:

1. Agents propose, people merge. An agent works on a branch and opens a pull request. It never pushes to the main branch, and its token does not have permission to.
2. Generated tests pass the chapter 2 checklist, including --repeat-each=5 locally before review.
3. No new skips without a person. The fixme/skip/only guard from chapter 4 runs on every pull request, alongside forbidOnly: !!process.env.CI.
4. Assertion changes need a reason. A diff that changes an expected value, removes an assertion or loosens a matcher links to the ticket or requirement that changed. Reviewers reject "the test was failing" as a reason.
5. No AI in the gating run. The suite that blocks a release runs committed code against real browsers and does not call a model.
6. Data rules apply to tools too. Only approved providers, and only artefacts that have been checked for secrets and personal data.

6. Reviewing agent diffs#

Reviewing generated code feels different from reviewing a colleague's. It is usually tidy, consistently formatted and confidently commented, and those are the signals reviewers use to skim. Read it as if it came from a capable new hire on their first day:

• Read the plan or prompt first, then the diff. Check that the code does what was asked, and nothing else.
• Sort lines into "finds things" and "judges things" (chapter 4). Spend your time on the second group.
• Distrust comments that explain why something is fine. In chapter 4, the healer's comment cited backup files we had left lying around as evidence.
• Run it. An unread diff with green CI is still unreviewed.

What this adds up to#

Used like this, AI makes a test suite cheaper to write and maintain, and it does not make the suite less trustworthy. The agent does the typing, the exploring, the locator fixes and the first read of a failure. People keep the requirements, the risk calls, the assertions and the merge button. That split is also what the certificate's practical task asks you to show: an agent-generated suite, reviewed by you, running green on real browsers.