AI Test Automation Quiz

1. Which use of AI in test automation keeps the run that gates a release fully deterministic?

When the AI works at authoring time, CI runs ordinary committed code: the same commit does the same thing every time. The other three options put a model's decision inside the run, where nobody reviews it.

2. Playwright MCP returns an accessibility snapshot after each action. Why do agents using it tend to produce good locators?

The agent sees "checkbox 'Toggle Todo'" rather than pixels or markup, so the natural way to refer to it is the role-based locator. The flip side: elements without accessible names are just as hard for the agent to tell apart as for a screen-reader user.

3. The Playwright planner wrote a plan for your app. Its summary admits it only looked at the start page. What does that tell you about the plan's "expect" lines?

Every expect line becomes an assertion. Expectations the planner did not observe are guesses, which can be right on a well-known demo and wrong on your app. The plan is the cheapest place to fix them, and the planner's honest summary is a good reason to look.

4. A generated test asserts toHaveCSS('text-decoration-line', 'line-through') for a completed todo. The requirement only says completed items are "marked as completed". What is the right review decision?

An assertion on styling breaks on a harmless redesign and trains the team to ignore failures. Assert the state the requirement describes (checked box, completed state, counter) and leave visual details to visual testing.

5. A healer agent's diff changes a button locator from "Add to cart" to "Add to bag" and leaves the test failing on a wrong total. How should you read that outcome?

Fixing what finds elements and leaving what judges results is exactly the right split. A healer that also "fixed" the total would have turned a pricing regression into a passing test.

6. After an unattended healer run, CI reports "1 skipped, 11 passed" and exits with code 0. The skip is new. What should have prevented this from looking like a green build?

forbidOnly only catches test.only. Playwright has no built-in check for newly added skips, so add one. Retries and repeated healing do not address a test that was turned off.

7. You want a model to triage a failed test. Which input most improves the chance it says correctly whether the app or the test is wrong?

The error shows that the test and the page disagree. Only the requirement says which one is right, and the diff says what changed. The trace adds detail and a lot of sensitive data, and it cannot settle what the correct behaviour is.

8. Your reviewed suite runs on TestingBot with two projects and two workers. Eight tests pass, and the dashboard shows four sessions. Why?

Sessions follow Playwright's workers, not individual tests. A worker that hits a failure is replaced, so a failed test is always the last in its session and its status is not overwritten by a later pass.

9. A teammate wants to give an exploring agent the production admin login "because staging data is unrealistic". What is the strongest objection?

An agent with production credentials can click "delete" as easily as "view", and page content can carry prompt injection. Limit what the agent can reach. Do not rely on the model to behave.