Exit course
Chapter 7 of 10 Intermediate 10 min read 2 quick checks Updated

Run Selenium C# tests on the TestingBot grid

On this page
  1. What changes and what doesn't
  2. Options and tb:options
  3. The final DriverFactory
  4. The 60-second trap
  5. Reporting pass or fail
  6. Run it
  7. Testing a site that is not public
  8. Debugging a remote failure

Everything so far ran on your own machine: one operating system, whichever browser versions you have installed, and as many parallel browsers as your CPU can take. A Selenium grid runs the browsers on remote machines instead. The tests stay exactly the same; only the driver factory changes, from new ChromeDriver() to new RemoteWebDriver(gridUrl, options). This chapter makes that switch behind an environment variable, using the TestingBot grid. The same RemoteWebDriver code works with a self-hosted Selenium Grid; only the URL and the vendor-specific options differ.

Everything in this chapter ran on TestingBot: the 16-test suite on Chrome 156 and Firefox 158 on Windows 11, the form tests on Safari 18 on macOS Sequoia, 40 sessions in all, every one reported as passed, plus a deliberate failure to check that the failed status and the failure message come through.

What changes and what doesn't#

Stays the same Changes
Tests, page objects, locators, waits, assertions How the driver is created
BROWSER, [TestCase], categories The browser and OS come from capabilities
Screenshot on failure You also get a video and logs of every session
[Parallelizable] (next chapter) The worker count is bounded by your plan's parallel limit

Because chapter 3 put driver creation in DriverFactory, the switch touches that class and a few lines of BaseTest.

Options and tb:options#

A remote session is requested with the same options classes you use locally (ChromeOptions, FirefoxOptions, EdgeOptions, SafariOptions), which all derive from DriverOptions. Two standard W3C capabilities are properties on it:

  • BrowserVersion, such as "latest" or a specific major version;
  • PlatformName, the operating system. On TestingBot these are names such as WIN11 for Windows 11 or SEQUOIA for macOS Sequoia.

Vendor settings go in a namespaced capability, added with AddAdditionalOption. TestingBot's is tb:options, which carries your key and secret, name (the test name shown in the dashboard) and build (which groups the tests of one run). The capabilities generator produces the exact C# for any browser and OS, and test options lists everything else tb:options accepts, such as screen resolution, time zone, or turning off video recording, which is on by default.

The final DriverFactory#

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Edge;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Safari;

namespace SeleniumCourse.Support;

public static class DriverFactory
{
    private static readonly Uri GridUrl = new("https://hub.testingbot.com/wd/hub");

    public static string Browser =>
        Environment.GetEnvironmentVariable("BROWSER") ?? "chrome";

    public static bool OnGrid =>
        Environment.GetEnvironmentVariable("GRID") == "testingbot";

    private static bool Headless =>
        Environment.GetEnvironmentVariable("HEADLESS") == "1";

    public static IWebDriver Create(string testName) =>
        OnGrid ? CreateRemote(testName) : CreateLocal();

    private static IWebDriver CreateLocal()
    {
        switch (Browser)
        {
            case "chrome":
                var chrome = new ChromeOptions();
                if (Headless) chrome.AddArguments("--headless=new", "--window-size=1920,1080");
                return new ChromeDriver(chrome);
            case "firefox":
                var firefox = new FirefoxOptions();
                if (Headless) firefox.AddArgument("-headless");
                return new FirefoxDriver(firefox);
            default:
                throw new ArgumentException($"BROWSER '{Browser}' is only configured for GRID=testingbot");
        }
    }

    private static IWebDriver CreateRemote(string testName)
    {
        DriverOptions options = Browser switch
        {
            "chrome" => new ChromeOptions(),
            "firefox" => new FirefoxOptions(),
            "edge" => new EdgeOptions(),
            "safari" => new SafariOptions(),
            _ => throw new ArgumentException($"Unknown BROWSER '{Browser}'"),
        };
        options.BrowserVersion = "latest";
        options.PlatformName = Browser == "safari" ? "SEQUOIA" : "WIN11";
        options.AddAdditionalOption("tb:options", new Dictionary<string, object>
        {
            ["key"] = Required("TB_KEY"),
            ["secret"] = Required("TB_SECRET"),
            ["name"] = testName,
            ["build"] = Environment.GetEnvironmentVariable("TB_BUILD") ?? "local",
        });

        // Outlast the grid's queue (up to 6 minutes): the default command timeout is 60 seconds.
        return new RemoteWebDriver(GridUrl, options.ToCapabilities(), TimeSpan.FromMinutes(7));
    }

    private static string Required(string name) =>
        Environment.GetEnvironmentVariable(name)
        ?? throw new InvalidOperationException($"Set the {name} environment variable");
}

What is new:

  • GRID=testingbot switches to CreateRemote. Without it the local workflow is unchanged.
  • Safari only exists on macOS, so it gets SEQUOIA; the others run on Windows 11. Edge and Safari are only configured for the grid, and asking for them locally fails with a clear message rather than silently starting Chrome.
  • Credentials come from TB_KEY and TB_SECRET, never from code. Find both in the TestingBot member area; in CI they become secrets. Required fails with a readable message when one is missing.
  • name is NUnit's test name including [TestCase] arguments, such as DropdownSelectionIsSubmitted("Two","2"), so each dashboard entry maps to one line of your test output. BaseTest passes it in: Driver = DriverFactory.Create(TestContext.CurrentContext.Test.Name);
  • build comes from TB_BUILD, so all sessions from one run are grouped.

The 60-second trap#

The last line uses the RemoteWebDriver constructor that takes a command timeout. The shorter new RemoteWebDriver(GridUrl, options) uses 60 seconds for every command, including the one that creates the session. Our first run while other jobs were using the same account failed in SetUp with:

OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL https://hub.testingbot.com/wd/hub/session timed out after 60 seconds.

The session request was waiting in the grid's queue, which is normal when all your parallel slots are busy: according to TestingBot's documentation, queued sessions wait for up to 6 minutes for a free slot. A client that gives up after 60 seconds turns a short queue into a failed test. The fix is a command timeout longer than that 6-minute queue window, so the client never gives up before the grid does; the sample uses 7 minutes. Our runs started once a slot was free and passed, and in one of them the first session waited 2 minutes 34 seconds in the queue before it started.

Grid runs pass in the morning, but in the afternoon, when other teams share the account, a few tests fail in SetUp with "timed out after 60 seconds" on /session. What do you change?

The session request is queued on the grid, and the client's default 60-second HTTP timeout gives up first. The constructor overload with a TimeSpan fixes it at the source. Retrying sends more requests into the same full queue, and WebDriverWait is not involved in creating the session at all. (By default NUnit's [Retry] also only retries assertion failures, not exceptions like this one.)

Reporting pass or fail#

The grid only sees WebDriver commands. It cannot know that an Assert.That failed; from its side the session simply ended. Unless you tell it, the dashboard cannot show which tests passed. The final BaseTest reports the outcome in [TearDown], before Quit():

using NUnit.Framework.Interfaces;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;

namespace SeleniumCourse.Support;

public abstract class BaseTest
{
    protected IWebDriver Driver { get; private set; } = null!;

    [SetUp]
    public void StartBrowser()
    {
        Driver = DriverFactory.Create(TestContext.CurrentContext.Test.Name);
    }

    [TearDown]
    public void StopBrowser()
    {
        if (Driver is null)
        {
            return; // the browser never started, so there is nothing to report or quit
        }
        bool passed = TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Passed;
        try
        {
            if (!passed)
            {
                try
                {
                    SaveScreenshot();
                }
                catch (WebDriverException e)
                {
                    // A crashed browser cannot take a screenshot; still report the result below.
                    TestContext.Out.WriteLine($"No screenshot: {e.Message}");
                }
            }
            if (DriverFactory.OnGrid)
            {
                // Tell TestingBot the outcome while the session is still open.
                string result = passed ? "passed" : "failed";
                ((IJavaScriptExecutor)Driver).ExecuteScript($"tb:test-result={result}");
                TestContext.Out.WriteLine(
                    $"TestingBot session {((RemoteWebDriver)Driver).SessionId}: {result}");
            }
        }
        finally
        {
            Driver.Quit();
        }
    }

    // SaveScreenshot as in chapter 4
}

TestingBot recognises tb: commands sent through the JavaScript executor and handles them on the grid instead of running them in the page. Annotating tests lists the others, such as tb:test-name=.... This needs no extra package, but it must run while the session is open.

The screenshot has its own try/catch: if the browser has crashed, taking the screenshot throws, and without the catch control would jump straight to finally, so the session would be left without a status. The Driver is null guard is there because of the queue trap above: when session creation throws, NUnit still runs [TearDown], and without the guard a NullReferenceException from Quit() is added to the real error.

The alternative is TestingBot's REST API, which works after the session has closed and can also send the failure message. A small helper with HttpClient:

using System.Net.Http.Headers;
using System.Text;

namespace SeleniumCourse.Support;

public static class TestingBotApi
{
    private static readonly HttpClient Http = CreateClient();

    private static HttpClient CreateClient()
    {
        var client = new HttpClient { BaseAddress = new Uri("https://api.testingbot.com/v1/") };
        string credentials = $"{Environment.GetEnvironmentVariable("TB_KEY")}:" +
                             $"{Environment.GetEnvironmentVariable("TB_SECRET")}";
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
            "Basic", Convert.ToBase64String(Encoding.ASCII.GetBytes(credentials)));
        return client;
    }

    public static async Task ReportAsync(string sessionId, bool passed, string? message = null)
    {
        var fields = new Dictionary<string, string> { ["test[success]"] = passed ? "1" : "0" };
        if (message is not null)
        {
            fields["test[status_message]"] = message;
        }

        using var response = await Http.PutAsync(
            $"tests/{sessionId}", new FormUrlEncodedContent(fields));
        response.EnsureSuccessStatusCode();
    }
}

Called from an async teardown after Quit(), with the session ID saved in SetUp:

[TearDown]
public async Task Stop()
{
    if (driver is null) return;
    driver.Quit();

    var result = TestContext.CurrentContext.Result;
    await TestingBotApi.ReportAsync(sessionId,
        result.Outcome.Status == TestStatus.Passed, result.Message);
}

We ran it against a deliberately failing assertion; the API afterwards returned success: false for the session, with NUnit's full Expected ... But was ... text as the status message on the dashboard. The NUnit guide shows the same call inline. Pick one mechanism; using both is harmless but redundant.

The dashboard shows no pass/fail for any test, although NUnit reports results correctly. The TearDown calls Driver.Quit() and then ExecuteScript("tb:test-result=failed") inside a try that swallows exceptions. What is wrong?

The executor command travels over the WebDriver session, so the session must still be open. After Quit() it throws, and the empty catch hides that. To report after closing, use the REST API with the saved session ID.

Run it#

export TB_KEY=your-key
export TB_SECRET=your-secret
export TB_BUILD="course-$(date +%Y%m%d-%H%M)"
export GRID=testingbot

BROWSER=chrome dotnet test
BROWSER=firefox dotnet test
BROWSER=safari dotnet test --filter "Category=form"

Our run of the 16-test suite on Chrome and Firefox, plus the 8 form tests on Safari, put all 40 sessions into one build, each with its name, status, video and logs. Every test prints its session ID, which is the link between a line in your test output and a session in the dashboard; --logger "console;verbosity=normal" shows it.

Testing a site that is not public#

A cloud browser cannot reach localhost:5000 or a staging server behind your firewall. TestingBot Tunnel solves this: start it on a machine that can reach the site, then point RemoteWebDriver at the tunnel instead of the public hub, by default http://localhost:4445/wd/hub when the tunnel runs on the same machine as the tests. The NUnit guide has the exact code and the tunnel documentation covers installation; the local website testing course goes into it in depth.

Debugging a remote failure#

Remote failures feel harder because you cannot watch the browser. In practice you have more evidence than locally: the screenshot from BaseTest still works (it is a WebDriver command, so it captures the remote browser), and the printed session ID leads to the video and logs of exactly that run. Most "only fails on the grid" problems turn out to be a slower page load that an explicit wait would have covered, a window size different from your laptop's, or a real browser difference you would never have seen running Chrome alone, which is the reason to run on several browsers in the first place.