Exit course
Chapter 4 of 10 Beginner 8 min read 2 quick checks Updated

Assertions, Verifications and Waits in Selenium IDE

On this page
  1. assert and verify
  2. Which commands wait, and for what
  3. The bug this table explains
  4. Content that appears later
  5. Avoid pause
  6. The smoke checks in the project

A recorded test only becomes useful when it checks something, and it only stays useful when those checks do not fail at random. This chapter covers the two families of checks, assert and verify, and which commands wait for the page and which do not. The first test of this course failed now and then until we fixed exactly that.

assert and verify#

Most check commands come in two versions:

Assert Verify Checks
assertText verifyText an element's visible text equals the value
assertTitle verifyTitle the page title
assertElementPresent verifyElementPresent an element exists
assertElementNotPresent verifyElementNotPresent an element does not exist
assertChecked / assertNotChecked verifyChecked / verifyNotChecked a checkbox state
assertSelectedValue / assertSelectedLabel verifySelectedValue / verifySelectedLabel the selected option of a <select>
assertValue verifyValue an input's value
assert verify a stored variable (chapter 4)

The difference is what happens when the check fails:

  • assert... stops the test. The remaining commands do not run.
  • verify... records the failure and carries on. The test is still reported as failed at the end, but every command after it runs.

We ran this test to see both:

Command Target Value
open /login
verifyTitle Not the title
echo Still running after the failed verify
assertTitle Not the title either
echo Never printed
info: ✓ Verify demo › open | /login
info: ✕ Verify demo › verifyTitle | Not the title: Actual value 'The Internet' did not match 'Not the title' during verifyTitle:Not the title:
info: echo: Still running after the failed verify
info: ✓ Verify demo › echo | Still running after the failed verify
info: ✕ Verify demo › assertTitle | Not the title either: Actual value 'The Internet' did not match 'Not the title either' during assertTitle:Not the title either:
info: Finished test Verify demo Failure

When to use which. Use assert when the next steps make no sense if the check failed: if the login did not work, there is no point clicking around the secure area. Use verify for several independent checks on one page, such as five labels on a settings screen, where you want to see all the wrong ones in one run rather than one per run.

Keep one consequence in mind: after a failed verify, everything that follows still runs, including any step that reports a result somewhere. Chapter 7 marks tests as passed on TestingBot with a step at the end of each test, and that step would run after a failed verify too. In tests that report their own result, make the checks that decide pass or fail asserts.

A checkout test logs in, adds an item and pays. Which check should be an assert rather than a verify?

If the login failed, the following steps fail too, with errors that hide the real cause. Stop at the first failure that makes the rest meaningless; use verify for independent details.

Which commands wait, and for what#

Web pages change after you act on them: a click starts a page load, a request fills in a list, a spinner disappears. The IDE does some waiting for you, but not all of it. This is what the version 4 runtime does, read from its source and confirmed in our runs:

Command Waits for Up to
click the element to exist and be visible the step timeout
type, select, check, storeText, assertText, assertChecked ... the element to exist the step timeout
assertElementPresent the element to exist the step timeout
assertElementNotPresent nothing: it checks once
assertTitle, verifyTitle nothing: it checks once
assertText the element to exist, not for its text to match the step timeout
waitForElementPresent, waitForElementVisible, waitForElementNotPresent, waitForElementNotVisible, waitForElementEditable what the name says the value you give, in milliseconds
waitForText the element's text to equal the value the step timeout

The step timeout is Step Timeout under Config > Project in the IDE. selenium-side-runner uses its own --timeout option instead, 15 seconds by default.

The bug this table explains#

Our first version of "Log in with valid credentials" checked the heading straight after the click:

Command Target Value
click css=.radius
assertText css=h2 Secure Area

It passed most of the time and sometimes failed with:

StaleElementReferenceError: stale element reference: stale element not found
  ... during assertText:css=h2:Secure Area

Both pages have an <h2>. The click starts the navigation, but assertText can find the login page's heading before it is replaced. By the time it reads the text, that element is gone. assertText waits for an element to exist, and an h2 did exist, just the wrong one.

The fix is to first wait for something that only the new page has:

Command Target Value
click css=.radius
assertElementPresent css=#flash.success
assertText css=h2 Secure Area

The login page has no success banner, so assertElementPresent waits until the secure page is there, and the heading check reads the right element. After the change the suite passed three runs in a row locally, and again on the grid in chapter 7.

The same thing happens after Logout, with a twist: the secure page and the login page both show a green #flash.success banner. There, the test waits for the text itself:

Command Target Value
click linkText=Logout
waitForText css=h2 Login Page

waitForText keeps reading the element until the text matches, and it copes with the element being replaced in the meantime.

After clicking "Save", the test runs assertText | css=.message | Saved. Sometimes it fails with "Actual value 'Saving...' did not match 'Saved'". What is the fix?

assertText only waits for the element, and the element exists while it still says "Saving...". waitForText waits until the text matches. A pause is a guess that is too long on fast runs and too short on slow ones; a longer step timeout changes nothing because the element was already found.

Content that appears later#

The practice site's /dynamic_loading/2 page adds "Hello World!" to the page about five seconds after you click Start. The test:

Command Target Value
open /dynamic_loading/2
click css=#start > button
waitForElementVisible css=#finish 10000
assertText css=#finish > h4 Hello World!

waitForElementVisible gets an explicit timeout of 10 seconds because the delay is part of the page's behaviour, not a glitch. It finishes as soon as the element shows up, so a fast run stays fast.

Avoid pause#

pause | 5000 stops the test for five seconds no matter what. It is tempting because it "fixes" a flaky step, but it makes every run slower and still fails on the day the server takes six seconds. Each pause in a suite is a wait that should have been a waitFor... command on the thing you are actually waiting for.

The flaky tests course covers timing problems in depth, with the same principle applied to coded tests.

The smoke checks in the project#

The other smoke tests use these checks:

Test Checks
Wrong password shows an error assertElementPresent css=#flash.error, then assertElementNotPresent css=#flash.success, then verifyTitle The Internet
Check both checkboxes assertNotChecked on the first box, check it, then assertChecked on both
Pick an option from the dropdown select id=dropdown label=Option 2, then assertSelectedValue id=dropdown 2 and verifySelectedLabel id=dropdown Option 2

Note the order in the first one. assertElementNotPresent checks once without waiting, so it comes after a check that waited for the error to appear. On its own, straight after the click, it would pass even if the page had not responded yet.