Assertions, Verifications and Waits in Selenium IDE
On this page
A recorded test only becomes useful when it checks something, and it only stays useful when those checks do not fail at random. This chapter covers the two families of checks, assert and verify, and which commands wait for the page and which do not. The first test of this course failed now and then until we fixed exactly that.
assert and verify#
Most check commands come in two versions:
| Assert | Verify | Checks |
|---|---|---|
assertText |
verifyText |
an element's visible text equals the value |
assertTitle |
verifyTitle |
the page title |
assertElementPresent |
verifyElementPresent |
an element exists |
assertElementNotPresent |
verifyElementNotPresent |
an element does not exist |
assertChecked / assertNotChecked
|
verifyChecked / verifyNotChecked
|
a checkbox state |
assertSelectedValue / assertSelectedLabel
|
verifySelectedValue / verifySelectedLabel
|
the selected option of a <select>
|
assertValue |
verifyValue |
an input's value |
assert |
verify |
a stored variable (chapter 4) |
The difference is what happens when the check fails:
-
assert...stops the test. The remaining commands do not run. -
verify...records the failure and carries on. The test is still reported as failed at the end, but every command after it runs.
We ran this test to see both:
| Command | Target | Value |
|---|---|---|
open |
/login |
|
verifyTitle |
Not the title |
|
echo |
Still running after the failed verify |
|
assertTitle |
Not the title either |
|
echo |
Never printed |
info: ✓ Verify demo › open | /login
info: ✕ Verify demo › verifyTitle | Not the title: Actual value 'The Internet' did not match 'Not the title' during verifyTitle:Not the title:
info: echo: Still running after the failed verify
info: ✓ Verify demo › echo | Still running after the failed verify
info: ✕ Verify demo › assertTitle | Not the title either: Actual value 'The Internet' did not match 'Not the title either' during assertTitle:Not the title either:
info: Finished test Verify demo Failure
When to use which. Use assert when the next steps make no sense if the check failed: if the login did not work, there is no point clicking around the secure area. Use verify for several independent checks on one page, such as five labels on a settings screen, where you want to see all the wrong ones in one run rather than one per run.
Keep one consequence in mind: after a failed verify, everything that follows still runs, including any step that reports a result somewhere. Chapter 7 marks tests as passed on TestingBot with a step at the end of each test, and that step would run after a failed verify too. In tests that report their own result, make the checks that decide pass or fail asserts.
A checkout test logs in, adds an item and pays. Which check should be an assert rather than a verify?
If the login failed, the following steps fail too, with errors that hide the real cause. Stop at the first failure that makes the rest meaningless; use verify for independent details.
Which commands wait, and for what#
Web pages change after you act on them: a click starts a page load, a request fills in a list, a spinner disappears. The IDE does some waiting for you, but not all of it. This is what the version 4 runtime does, read from its source and confirmed in our runs:
| Command | Waits for | Up to |
|---|---|---|
click |
the element to exist and be visible | the step timeout |
type, select, check, storeText, assertText, assertChecked ... |
the element to exist | the step timeout |
assertElementPresent |
the element to exist | the step timeout |
assertElementNotPresent |
nothing: it checks once | |
assertTitle, verifyTitle
|
nothing: it checks once | |
assertText |
the element to exist, not for its text to match | the step timeout |
waitForElementPresent, waitForElementVisible, waitForElementNotPresent, waitForElementNotVisible, waitForElementEditable
|
what the name says | the value you give, in milliseconds |
waitForText |
the element's text to equal the value | the step timeout |
The step timeout is Step Timeout under Config > Project in the IDE. selenium-side-runner uses its own --timeout option instead, 15 seconds by default.
The bug this table explains#
Our first version of "Log in with valid credentials" checked the heading straight after the click:
| Command | Target | Value |
|---|---|---|
click |
css=.radius |
|
assertText |
css=h2 |
Secure Area |
It passed most of the time and sometimes failed with:
StaleElementReferenceError: stale element reference: stale element not found
... during assertText:css=h2:Secure Area
Both pages have an <h2>. The click starts the navigation, but assertText can find the login page's heading before it is replaced. By the time it reads the text, that element is gone. assertText waits for an element to exist, and an h2 did exist, just the wrong one.
The fix is to first wait for something that only the new page has:
| Command | Target | Value |
|---|---|---|
click |
css=.radius |
|
assertElementPresent |
css=#flash.success |
|
assertText |
css=h2 |
Secure Area |
The login page has no success banner, so assertElementPresent waits until the secure page is there, and the heading check reads the right element. After the change the suite passed three runs in a row locally, and again on the grid in chapter 7.
The same thing happens after Logout, with a twist: the secure page and the login page both show a green #flash.success banner. There, the test waits for the text itself:
| Command | Target | Value |
|---|---|---|
click |
linkText=Logout |
|
waitForText |
css=h2 |
Login Page |
waitForText keeps reading the element until the text matches, and it copes with the element being replaced in the meantime.
After clicking "Save", the test runs assertText | css=.message | Saved. Sometimes it fails with "Actual value 'Saving...' did not match 'Saved'". What is the fix?
assertText only waits for the element, and the element exists while it still says "Saving...". waitForText waits until the text matches. A pause is a guess that is too long on fast runs and too short on slow ones; a longer step timeout changes nothing because the element was already found.
Content that appears later#
The practice site's /dynamic_loading/2 page adds "Hello World!" to the page about five seconds after you click Start. The test:
| Command | Target | Value |
|---|---|---|
open |
/dynamic_loading/2 |
|
click |
css=#start > button |
|
waitForElementVisible |
css=#finish |
10000 |
assertText |
css=#finish > h4 |
Hello World! |
waitForElementVisible gets an explicit timeout of 10 seconds because the delay is part of the page's behaviour, not a glitch. It finishes as soon as the element shows up, so a fast run stays fast.
Avoid pause#
pause | 5000 stops the test for five seconds no matter what. It is tempting because it "fixes" a flaky step, but it makes every run slower and still fails on the day the server takes six seconds. Each pause in a suite is a wait that should have been a waitFor... command on the thing you are actually waiting for.
The flaky tests course covers timing problems in depth, with the same principle applied to coded tests.
The smoke checks in the project#
The other smoke tests use these checks:
| Test | Checks |
|---|---|
| Wrong password shows an error |
assertElementPresent css=#flash.error, then assertElementNotPresent css=#flash.success, then verifyTitle The Internet
|
| Check both checkboxes |
assertNotChecked on the first box, check it, then assertChecked on both |
| Pick an option from the dropdown |
select id=dropdown label=Option 2, then assertSelectedValue id=dropdown 2 and verifySelectedLabel id=dropdown Option 2
|
Note the order in the first one. assertElementNotPresent checks once without waiting, so it comes after a check that waited for the error to appear. On its own, straight after the click, it would pass even if the page had not responded yet.