NUnit assertions and screenshots on failure
On this page
A browser test is only as useful as its failure message. This chapter covers NUnit's constraint model, checking several things at once without stopping at the first, and saving a screenshot whenever a test fails.
The constraint model#
Every assertion in NUnit 4 and 5 is written the same way: Assert.That(actual, constraint).
Assert.That(driver.Title, Is.EqualTo("Web form"));
Assert.That(heading.Text, Does.StartWith("Web").IgnoreCase);
Assert.That(driver.Url, Does.Contain("my-select=2"));
Assert.That(button.Enabled, Is.True);
Assert.That(driver.FindElements(By.Id("error")), Is.Empty);
Assert.That(rows, Has.Count.EqualTo(3));
Assert.That(price, Is.GreaterThan(0).And.LessThan(100));
Assert.That(options, Does.Contain("Two"));
The actual value always comes first. The constraint reads like a sentence and builds the failure message for you:
Assert.That("Web form", Is.EqualTo("Web Form"))
String lengths are both 8. Strings differ at index 4.
Expected: "Web Form"
But was: "Web form"
---------------^
NUnit 4 moved the older classic assertions (Assert.AreEqual(expected, actual) and friends) to ClassicAssert in the NUnit.Framework.Legacy namespace. They still work, but the analyzer flags each one with NUnit2005, "Consider using the constraint model", and the expected-first argument order is a classic source of backwards failure messages. Use Assert.That in new code.
The analyzer catches real mistakes here too. Assert.That(new[] { 1, 2 }, Has.Count.EqualTo(3)) does not compile:
error NUnit2022: The type of the actual argument - 'int[]' - has no property 'Count'
Arrays have Length, lists and Selenium's IReadOnlyCollection have Count. Without the analyzer that would have been a confusing runtime failure.
A failing test reports Expected: "Received!" But was: "Form submission", yet the page clearly shows "Received!". What is the most likely mistake?
In the constraint model "But was" is always the actual value, so the test read some other element's text, here a heading. That unambiguity is a reason to prefer it over the classic expected-first methods, where swapped arguments produce exactly this kind of confusing message. Is.EqualTo is case-sensitive unless you add .IgnoreCase.
Several checks in one test#
A test that checks a page's state often wants every difference, not just the first. Wrap the assertions in a multiple-assert scope:
[Test]
public void FieldsStartWithTheirDefaults()
{
using (Assert.EnterMultipleScope())
{
Assert.That(Driver.FindElement(By.Name("my-readonly")).GetDomProperty("value"),
Is.EqualTo("Readonly input"));
Assert.That(Driver.FindElement(By.Id("my-check-1")).Selected, Is.True);
Assert.That(Driver.FindElement(By.Id("my-check-2")).Selected, Is.False);
Assert.That(Driver.FindElement(By.Name("my-range")).GetDomProperty("value"),
Is.EqualTo("5"));
}
}
All four assertions run, and the test fails at the end of the using block if any of them did. When we changed two of the expectations on purpose, one run reported both:
Multiple failures or warnings in test:
1) Assert.That(Driver.FindElement(By.Id("my-check-2")).Selected, Is.True)
Expected: True
But was: False
2) Assert.That(Driver.FindElement(By.Name("my-range")).GetDomProperty("value"), Is.EqualTo("7"))
String lengths are both 1. Strings differ at index 0.
Expected: "7"
But was: "5"
-----------^
Assert.EnterMultipleScope() arrived in NUnit 4.2. The older form, Assert.Multiple(() => { ... }), still works and does the same with a lambda. Note what does not get collected: an exception such as NoSuchElementException thrown inside the block still ends the test at once. Make sure the page is in the right state (with a wait or a plain assertion) before the block, and use the block for the details.
A screenshot for every failure#
The most useful artifact for a failed browser test is a picture of the browser at that moment. [TearDown] runs after every test, and TestContext tells it whether the test failed. Update Support/BaseTest.cs:
using NUnit.Framework.Interfaces;
using OpenQA.Selenium;
namespace SeleniumCourse.Support;
public abstract class BaseTest
{
protected IWebDriver Driver { get; private set; } = null!;
[SetUp]
public void StartBrowser()
{
Driver = DriverFactory.Create();
}
[TearDown]
public void StopBrowser()
{
try
{
if (TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Failed)
{
SaveScreenshot();
}
}
finally
{
Driver.Quit();
}
}
private void SaveScreenshot()
{
string name = string.Concat(TestContext.CurrentContext.Test.Name
.Select(c => char.IsLetterOrDigit(c) ? c : '_'));
string path = Path.Combine(TestContext.CurrentContext.WorkDirectory, $"{name}.png");
((ITakesScreenshot)Driver).GetScreenshot().SaveAsFile(path);
TestContext.AddTestAttachment(path, "Browser at the moment of failure");
}
}
Three details matter:
-
try/finally. If taking the screenshot throws (the browser crashed, say), the browser is still quit. Without it, a failing screenshot leaks a browser process on every failure. -
The file name is built from the test name with everything except letters and digits replaced, because
[TestCase]names contain quotes and commas that are not valid in file names on every OS. -
TestContext.AddTestAttachmentregisters the file with the test result. Test reports and IDEs show it next to the failure.
Test results files#
The console output is gone when the terminal closes. A TRX file is the standard .NET test result format that Visual Studio, Azure DevOps and most CI report tools read:
dotnet test --logger "trx;LogFileName=results.trx"
Results File: .../SeleniumCourse/TestResults/results.trx
When a test fails, its screenshot is copied into the TestResults folder next to the TRX file and linked from the failed result, so uploading that one folder from CI gives you the report and the pictures. Chapter 8 does exactly that.
The TearDown takes a screenshot and then calls Driver.Quit(), without try/finally. After a night of CI runs the agent has dozens of orphaned Chrome processes. Why?
The screenshot call talks to the browser and can fail exactly when things are already broken. A finally block guarantees Quit(). NUnit does run TearDown after failures, and Quit() ends the whole session; Close() is the one that closes a single window.
The tests still contain raw locators and navigation details. The next chapter moves those into page objects.