Debugging Cypress Tests
On this page
A failing test is only useful if you can find out why it failed. Cypress gives you more to work with than most tools: a command log with a snapshot of the page at every step, screenshots and videos from headless runs, and, when the suite runs on TestingBot, a video and logs for every session. This chapter walks through each, then follows a real flaky test from this course to its cause.
Time travel in the command log#
Run a spec with npx cypress open. The command log on the left lists every command as it ran, with its subject and how long it took.
- Hover over a command and the application frame shows the page as it was at that moment. Elements the command acted on are highlighted.
-
Click a command to pin that snapshot, and look at the browser's developer tools console: Cypress prints what the command yielded (for
cy.get, the elements it found, how many, and the selector). - Commands that change the page, such as
.click()and.type(), have a before and after snapshot you can switch between. - A failed command is red, and the error message underneath links to the line of your spec that queued it.
Most failures are answered here in under a minute: the snapshot shows the selector matched nothing, matched two things, or matched an element that was covered by a cookie banner.
.click() fails with "this element is being covered by another element". What is the quickest way to see what is covering it?
The snapshot shows the page at the moment of the failure, and the message says which element received the click. force: true skips the check that just found a real problem (a user could not click it either), and a sleep only helps if the overlay happens to disappear on its own.
Pausing and inspecting#
Two commands help when the snapshot is not enough:
it('pauses, debugs and takes a screenshot', () => {
cy.visit('/todo')
cy.get('[data-test=new-todo]').type('Buy milk{enter}')
cy.pause()
cy.get('.todo-list li').last().debug().should('have.text', 'Buy milk')
cy.screenshot('after-adding-a-todo')
})
-
cy.pause()stops the queue incypress open, so you can interact with the app and then step through the remaining commands one at a time. Incypress runit is ignored. -
.debug()triggers the browser debugger with the current subject available in the console when developer tools are open, and passes the subject through unchanged. -
cy.screenshot(name)saves a screenshot at that point, in both modes.
Our article on how to debug Cypress tests goes further into the debugger, console.log inside .then() and reading stack traces.
Screenshots and videos from headless runs#
When nobody is watching, cypress run keeps evidence for you:
-
Screenshots on failure are on by default (
screenshotOnRunFailure). They land incypress/screenshots/<spec>/, one per failed attempt, named after the test, for examplea flaky test -- completes the default Walk the dog todo (failed) (attempt 2).png. -
Videos have been off by default since Cypress 13. Turn them on with
video: truein the config or--config video=truefor one run; each spec then producescypress/videos/<spec>.mp4.
In CI, upload both folders as build artifacts when the job fails, or they disappear with the runner.
Debugging a run on TestingBot#
When the suite runs on TestingBot (chapter 8), every browser configuration is a session in the build you named, and the session page in the dashboard has a video of the whole run and the Cypress logs from the virtual machine. The CLI prints the build link when the run starts. Because the run happened on a different operating system and browser version than your laptop, check the session's configuration before trying to reproduce a failure locally: a failure only on Firefox on Windows needs that browser, not a rerun in Electron.
A spec passes locally in Chrome but fails on TestingBot only in the Firefox session. What should you look at first?
The session's video and log show what Firefox actually rendered. A browser-specific failure is often a real bug your users would hit; hiding it with a longer timeout or by dropping the browser throws that information away.
A real flaky test, from start to finish#
While writing chapter 2 we had a test that completed one of the two todos the demo app creates on first load:
it('completes the default "Walk the dog" todo', () => {
cy.visit('/todo')
cy.contains('.todo-list li', 'Walk the dog').find('.toggle').check()
cy.contains('.todo-list li', 'Walk the dog').should('have.class', 'completed')
})
With retries: { runMode: 2 } it sometimes passed on its second attempt, and the summary still said the run passed. With retries turned off (--config retries=0) it failed most of the time. The failure screenshot showed the checkbox ticked, but the "completed" style on the other todo, "Pay electric bill".
That is not a timing problem, so no amount of waiting would fix it. A second test logged the ids the app gives its rows:
it('shows why: both default todos share one id', () => {
cy.visit('/todo')
cy.get('.todo-list li').then(($li) => {
const ids = [...$li].map((li) => li.dataset.id)
cy.task('log', `ids: ${ids.join(', ')}`)
})
})
Three runs printed:
ids: 1794498593295, 1794498593295
ids: 1794498620923, 1794498620924
ids: 1794498648166, 1794498648166
The demo app uses the current time in milliseconds as each todo's id, and it creates both default todos in the same millisecond more often than not. With two rows sharing an id, toggling the second one updates the first. The run where the ids differed by one millisecond is the kind of run in which the test passed.
Three lessons that apply to any suite:
- A pass on retry is a symptom, not a fix. Here, retries turned a real application bug into an occasional green run.
- Look at the evidence before changing the test. The screenshot showed the wrong row changing, which pointed away from timing straight away.
- Write the test around the behaviour you care about. The course tests now create their own todos (as in chapters 2 and 3), whose ids are a few milliseconds apart, so they test toggling without depending on the bug. In a real project you would also file the bug.
A test fails about one run in three with an assertion error, and passes on retry. The failure screenshot shows the action applied to the wrong row. What is the most useful next step?
Timing problems show up as "not found yet" or "not visible yet". An action that lands on the wrong element is a different symptom, and both more retries and a sleep would only hide it.