Skip to main content

Reqnroll Automated App Testing

Reqnroll is an open-source .NET test automation framework for Behavior-Driven Development (BDD). It is the community-maintained successor to SpecFlow and is fully compatible with Gherkin syntax.

Reqnroll works with all major .NET versions (.NET 6.0, 7.0, 8.0, 9.0) and integrates with popular test frameworks:

  • NUnit
  • xUnit
  • MSTest
  • TUnit

Migrating from SpecFlow? Reqnroll is based on the SpecFlow codebase and provides an easy migration path. See the migration section below.

In the example below, we run a Gherkin scenario against the TestingBot demo app on both Android and iOS. The app adds two numbers and shows the sum.

Installation

First, install the Reqnroll for Visual Studio 2022 extension from the Visual Studio Marketplace to get syntax highlighting, navigation and project templates.

Create a new project

Create a new test project and add the required packages:

# Create a new NUnit test project
dotnet new nunit -n AppiumReqnrollTests
cd AppiumReqnrollTests

# Add Reqnroll with NUnit integration
dotnet add package Reqnroll.NUnit

# Add the Appium .NET client
dotnet add package Appium.WebDriver
dotnet add package Selenium.WebDriver
dotnet add package Selenium.Support

Or add these packages to your .csproj file:

<ItemGroup>
  <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
  <PackageReference Include="NUnit" Version="4.1.0" />
  <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
  <PackageReference Include="Reqnroll.NUnit" Version="2.3.0" />
  <PackageReference Include="Appium.WebDriver" Version="6.1.0" />
  <PackageReference Include="Selenium.WebDriver" Version="4.27.0" />
  <PackageReference Include="Selenium.Support" Version="4.27.0" />
</ItemGroup>

Alternative packages for other test frameworks:

  • Reqnroll.xUnit - for xUnit 2.x
  • Reqnroll.xUnit.v3 - for xUnit 3.x
  • Reqnroll.MsTest - for MSTest
  • Reqnroll.TUnit - for TUnit

Your first Reqnroll test

Reqnroll tests consist of three parts: a Feature file (Gherkin), Step Definitions (C#), and a Driver helper class.

1. Feature File

Create a Features folder and add Calculator.feature:

Feature: Calculator

Scenario Outline: Can do simple math
    Given I am using the calculator
    When I add inputA for "5"
    When I add inputB for "10"
    Then I should see the sum "15"

2. Driver Helper Class

Create a helper class to manage the Appium connection to TestingBot:

using OpenQA.Selenium;
using OpenQA.Selenium.Appium;
using OpenQA.Selenium.Appium.Android;

namespace AppiumReqnrollTests.Support;

public class TestingBotDriver : IDisposable
{
    public AndroidDriver Driver { get; private set; }
    private bool _testPassed = true;

    public void Initialize()
    {
        var options = new AppiumOptions
        {
            PlatformName = "Android",
            AutomationName = "UiAutomator2",
            App = "https://testingbot.com/appium/sample.apk"
        };
        options.AddAdditionalAppiumOption("appium:deviceName", "Galaxy S24");
        options.AddAdditionalAppiumOption("appium:platformVersion", "14.0");
        options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
        {
            ["key"] = Environment.GetEnvironmentVariable("TB_KEY") ?? "",
            ["secret"] = Environment.GetEnvironmentVariable("TB_SECRET") ?? "",
            ["name"] = "Reqnroll Appium Test",
            ["realDevice"] = true
        });

        Driver = new AndroidDriver(
            new Uri("https://hub.testingbot.com/wd/hub"),
            options,
            TimeSpan.FromSeconds(120)
        );
    }

    public void MarkTestFailed() => _testPassed = false;

    public void Dispose()
    {
        if (Driver != null)
        {
            var status = _testPassed ? "passed" : "failed";
            ((IJavaScriptExecutor)Driver).ExecuteScript($"tb:test-result={status}");
            Driver.Dispose();
        }
    }
}

3. Step Definitions

Create StepDefinitions/CalculatorSteps.cs:

using OpenQA.Selenium.Appium;
using Reqnroll;
using AppiumReqnrollTests.Support;

namespace AppiumReqnrollTests.StepDefinitions;

[Binding]
public class CalculatorSteps
{
    private readonly TestingBotDriver _tbDriver;

    public CalculatorSteps(TestingBotDriver tbDriver)
    {
        _tbDriver = tbDriver;
    }

    [Given("I am using the calculator")]
    public void GivenIAmUsingTheCalculator()
    {
        if (_tbDriver.Driver == null)
            _tbDriver.Initialize();
    }

    [When(@"I add inputA for ""(.*)""")]
    public void WhenIAddInputA(string amount)
    {
        var inputA = _tbDriver.Driver.FindElement(AppiumBy.AccessibilityId("inputA"));
        inputA.SendKeys(amount);
    }

    [When(@"I add inputB for ""(.*)""")]
    public void WhenIAddInputB(string amount)
    {
        var inputB = _tbDriver.Driver.FindElement(AppiumBy.AccessibilityId("inputB"));
        inputB.SendKeys(amount);
    }

    [Then(@"I should see the sum ""(.*)""")]
    public void ThenIShouldSeeTheSum(string sum)
    {
        var sumElement = _tbDriver.Driver.FindElement(AppiumBy.AccessibilityId("sum"));
        Assert.That(sumElement.Text, Is.EqualTo(sum));
    }
}

4. Hooks for Setup/Teardown

Create Hooks/TestHooks.cs to manage the driver lifecycle:

using Reqnroll;
using AppiumReqnrollTests.Support;

namespace AppiumReqnrollTests.Hooks;

[Binding]
public class TestHooks
{
    private readonly TestingBotDriver _tbDriver;

    public TestHooks(TestingBotDriver tbDriver)
    {
        _tbDriver = tbDriver;
    }

    [AfterScenario]
    public void AfterScenario(ScenarioContext scenarioContext)
    {
        if (scenarioContext.TestError != null)
        {
            _tbDriver.MarkTestFailed();
        }
        _tbDriver.Dispose();
    }
}

Run your tests with:

TB_KEY=your_key TB_SECRET=your_secret dotnet test

Real Device Testing

The example below runs the Reqnroll scenario on a real Android device. Set realDevice: true inside tb:options to target physical hardware. 

using OpenQA.Selenium;
using OpenQA.Selenium.Appium;
using OpenQA.Selenium.Appium.Android;

namespace AppiumReqnrollTests.Support;

public class TestingBotDriver : IDisposable
{
    public AndroidDriver Driver { get; private set; }
    private bool _testPassed = true;

    public void Initialize()
    {
        var options = new AppiumOptions
        {
            PlatformName = "Android",
            AutomationName = "UiAutomator2",
            App = "https://testingbot.com/appium/sample.apk"
        };
        options.AddAdditionalAppiumOption("appium:deviceName", "Galaxy S24");
        options.AddAdditionalAppiumOption("appium:platformVersion", "14.0");
        options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
        {
            ["key"] = Environment.GetEnvironmentVariable("TB_KEY") ?? "",
            ["secret"] = Environment.GetEnvironmentVariable("TB_SECRET") ?? "",
            ["name"] = "Reqnroll Android Real Device",
            ["realDevice"] = true
        });

        Driver = new AndroidDriver(
            new Uri("https://hub.testingbot.com/wd/hub"),
            options,
            TimeSpan.FromSeconds(120)
        );
    }

    public void MarkTestFailed() => _testPassed = false;

    public void Dispose()
    {
        if (Driver != null)
        {
            var status = _testPassed ? "passed" : "failed";
            ((IJavaScriptExecutor)Driver).ExecuteScript($"tb:test-result={status}");
            Driver.Dispose();
        }
    }
}

The example below runs the Reqnroll scenario on a real iPhone. Use IOSDriver instead of AndroidDriver. 

using OpenQA.Selenium;
using OpenQA.Selenium.Appium;
using OpenQA.Selenium.Appium.iOS;

namespace AppiumReqnrollTests.Support;

public class TestingBotDriver : IDisposable
{
    public IOSDriver Driver { get; private set; }
    private bool _testPassed = true;

    public void Initialize()
    {
        var options = new AppiumOptions
        {
            PlatformName = "iOS",
            AutomationName = "XCUITest",
            App = "https://testingbot.com/appium/sample.ipa"
        };
        options.AddAdditionalAppiumOption("appium:deviceName", "iPhone 15");
        options.AddAdditionalAppiumOption("appium:platformVersion", "18.0");
        options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
        {
            ["key"] = Environment.GetEnvironmentVariable("TB_KEY") ?? "",
            ["secret"] = Environment.GetEnvironmentVariable("TB_SECRET") ?? "",
            ["name"] = "Reqnroll iOS Real Device",
            ["realDevice"] = true
        });

        Driver = new IOSDriver(
            new Uri("https://hub.testingbot.com/wd/hub"),
            options,
            TimeSpan.FromSeconds(300)
        );
    }

    public void MarkTestFailed() => _testPassed = false;

    public void Dispose()
    {
        if (Driver != null)
        {
            var status = _testPassed ? "passed" : "failed";
            ((IJavaScriptExecutor)Driver).ExecuteScript($"tb:test-result={status}");
            Driver.Dispose();
        }
    }
}

Simulator/Emulator Testing

Run the same Reqnroll scenario on an Android emulator. Omit realDevice from tb:options to target an emulator. 

var options = new AppiumOptions
{
    PlatformName = "Android",
    AutomationName = "UiAutomator2",
    App = "https://testingbot.com/appium/sample.apk"
};
options.AddAdditionalAppiumOption("appium:deviceName", "Pixel 8");
options.AddAdditionalAppiumOption("appium:platformVersion", "14.0");
options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
{
    ["key"] = Environment.GetEnvironmentVariable("TB_KEY") ?? "",
    ["secret"] = Environment.GetEnvironmentVariable("TB_SECRET") ?? "",
    ["name"] = "Reqnroll Android Emulator"
});

Driver = new AndroidDriver(
    new Uri("https://hub.testingbot.com/wd/hub"),
    options,
    TimeSpan.FromSeconds(240)
);

Run the same Reqnroll scenario on an iOS simulator. 

var options = new AppiumOptions
{
    PlatformName = "iOS",
    AutomationName = "XCUITest",
    App = "https://testingbot.com/appium/sample.zip"
};
options.AddAdditionalAppiumOption("appium:deviceName", "iPhone 15");
options.AddAdditionalAppiumOption("appium:platformVersion", "17.4");
options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
{
    ["key"] = Environment.GetEnvironmentVariable("TB_KEY") ?? "",
    ["secret"] = Environment.GetEnvironmentVariable("TB_SECRET") ?? "",
    ["name"] = "Reqnroll iOS Simulator"
});

Driver = new IOSDriver(
    new Uri("https://hub.testingbot.com/wd/hub"),
    options,
    TimeSpan.FromSeconds(300)
);

Uploading Your App

Please see our Upload Mobile App documentation to find out how to upload your app to TestingBot for testing.

Specify Browsers & Devices

To run your existing tests on TestingBot, your tests will need to be configured to use the TestingBot remote machines. If the test was running on your local machine or network, you can simply change your existing test like this:

Before: 
Driver = new IOSDriver(
    new Uri("http://localhost:4723/wd/hub"),
    options
);
After: 
Driver = new IOSDriver(
    new Uri("https://hub.testingbot.com/wd/hub"),
    options
);

Configuring Capabilities

Capabilities are key-value pairs that allow you to customize your tests on TestingBot.
Appium provides its own set of capabilities which you can specify.
TestingBot also provides its own Custom Capabilities, to customize tests run on the TestingBot platform.

You can use the drop-down menus below to see how to configure your tests to run on a specific mobile device:

Testing Internal Websites

We've built TestingBot Tunnel, to provide you with a secure way to run tests against your staged or internal webapps.
Please see our TestingBot Tunnel documentation for more information about this easy to use tunneling solution.

The example below shows how to easily run a Reqnroll test with our Tunnel:

1. Download our tunnel and start the tunnel: 

java -jar testingbot-tunnel.jar key secret

2. Adjust your test: instead of pointing to 'hub.testingbot.com/wd/hub' like the example above, change it to point to your tunnel's IP address.
Assuming you run the tunnel on the same machine you run your tests, change to 'localhost:4445/wd/hub'. localhost is the machine running the tunnel, 4445 is the default port of the tunnel.

This way your test will go securely through the tunnel to TestingBot and back:

var options = new AppiumOptions
{
    PlatformName = "Android",
    AutomationName = "UiAutomator2",
    App = "tb://..."
};
options.AddAdditionalAppiumOption("appium:deviceName", "Pixel 8");
options.AddAdditionalAppiumOption("appium:platformVersion", "14.0");
options.AddAdditionalAppiumOption("tb:options", new Dictionary<string, object>
{
    ["name"] = "Reqnroll Tunnel Test"
});

// Point to the local tunnel instead of hub.testingbot.com
Driver = new AndroidDriver(
    new Uri("http://localhost:4445/wd/hub"),
    options
);

Run tests in Parallel

Parallel Testing means running the same test, or multiple tests, simultaneously. This greatly reduces your total testing time.

Reqnroll supports parallel test execution when using NUnit or xUnit as the test runner.

NUnit Parallel Execution

Add to your AssemblyInfo.cs or any file:

using NUnit.Framework;

[assembly: Parallelizable(ParallelScope.Fixtures)]
[assembly: LevelOfParallelism(5)]

Configure in reqnroll.json:

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
  "bindingCulture": {
    "name": "en-US"
  }
}

xUnit Parallel Execution

Add xunit.runner.json:

{
  "parallelizeAssembly": true,
  "parallelizeTestCollections": true,
  "maxParallelThreads": 5
}

Queuing

Every TestingBot plan has a limit on parallel tests. If you exceed this limit, additional tests are queued (up to 6 minutes) and run as slots become available.

Mark tests as passed/failed

Report test results to TestingBot using JavaScript execution in your [AfterScenario] hook: 

[Binding]
public class TestHooks
{
    private readonly TestingBotDriver _tbDriver;

    public TestHooks(TestingBotDriver tbDriver)
    {
        _tbDriver = tbDriver;
    }

    [AfterScenario]
    public void AfterScenario(ScenarioContext scenarioContext)
    {
        if (_tbDriver.Driver != null)
        {
            var status = scenarioContext.TestError == null ? "passed" : "failed";
            ((IJavaScriptExecutor)_tbDriver.Driver).ExecuteScript($"tb:test-result={status}");
            _tbDriver.Driver.Dispose();
        }
    }
}

Migrating from SpecFlow

Reqnroll is designed as a drop-in replacement for SpecFlow. Most Appium projects can migrate with minimal changes:

1. Update NuGet Packages

# Remove SpecFlow packages
dotnet remove package SpecFlow
dotnet remove package SpecFlow.NUnit

# Add Reqnroll packages
dotnet add package Reqnroll.NUnit

2. Update Namespaces

Replace TechTalk.SpecFlow with Reqnroll in your code:

// Before (SpecFlow)
using TechTalk.SpecFlow;

// After (Reqnroll)
using Reqnroll;

3. Update Configuration

Rename specflow.json to reqnroll.json and update the schema reference:

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json"
}

4. Update Visual Studio Extension

Install the "Reqnroll for Visual Studio 2022" extension and uninstall the SpecFlow extension.

For a complete migration guide, see the official SpecFlow to Reqnroll migration guide.

Preparing your App

You do not need to install any code or plugin to run a test.
Below are some things that are necessary to successfully run a mobile test:

For Android:
  • Please supply the URL to your .apk or .aab file.
    Important: the .apk file needs to be a x86 build (x86 ABI) for Android emulators.
For iOS Real Device:
  • Please supply the URL to an .ipa file.
For iOS Simulator:
  • Please supply the URL to a .zip file that contains your .app
  • The .app needs to be an iOS Simulator build:
    • Create a Simulator build:
      xcodebuild -sdk iphonesimulator -configuration Debug
    • Zip the .app bundle:
      zip -r MyApp.zip MyApp.app

Additional Options

Appium provides a lot of options to configure your test.

Some important options that might help:

For Android:
  • appActivity and appPackage: by default, Appium will try to extract the main Activity from your apk. If this fails, you can supply your own with these options.
  • chromeOptions: additional chromedriver options you can supply.
  • otherApps: a JSON array of other apps that need to be installed, in URL format.
For Android & iOS:
  • locale: the language of the simulator/device (ex: fr_CA)

    This sets the locale on the entire device/simulator, except on physical iOS devices. For real iOS devices, we will pass -AppleLocale as argument to the app.

  • newCommandTimeout: How long (in seconds) Appium will wait for a new command from the client before assuming the client quit and ending the session
Was this page helpful?

Looking for More Help?

Have questions or need more information?
You can reach us via the following channels:

logo TestingBot