NodeJS Automated Testing with WebdriverIO

With WebDriverIO you can run Mocha, Jasmine (v2.0) and Cucumber tests.

See our WebdriverIO examples repository for some examples on how to use WebdriverIO with TestingBot. TestingBot has its own WebDriverIO service plugin which makes configuring your tests easier and which sends test meta-data back to TestingBot.

Let's start with making sure webdriverio is installed:

npm i --save-dev @wdio/cli @wdio/local-runner

WebDriverIO TestingBot Service

WebDriverIO contains a TestingBot service which makes it easy to run tests with WebDriver.io on TestingBot. With this service, WebDriverIO will start a TestingBot Tunnel if required, and send back test meta-data (passed state, name, build-id, ...)

In order to use the service you need to install it:

npm install @wdio/testingbot-service --save-dev

Cucumber Example

To run Cucumber tests on TestingBot, please follow these steps:

npm install @wdio/testingbot-service --save-dev
npm install @wdio/dot-reporter --save-dev
npm install @wdio/cucumber-framework --save-dev

Now we'll make a simple Cucumber test which uses Firefox on TestingBot to go to Google.com and verify the page's title.

Please create the following files:

wdio.conf.js

exports.config = {

	/**
	 * specify test files
	 */
	specs: [
		'./features/*.feature'
	],

	/**
	 * capabilities
	 */
	capabilities: [{
		browserName: 'chrome',
		browserVersion: 'latest',
		platformName: 'WIN10'
	}],

	/**
	 * test configurations
	 */
	logLevel: 'silent',
	coloredLogs: true,
	screenshotPath: 'screenshots',
	waitforTimeout: 10000,
	connectionRetryTimeout: 480000,
	framework: 'cucumber',
	services: [
        ['testingbot']
    ],
	user: 'api_key',
	key: 'api_secret',

	reporters: ['dot'],
	reporterOptions: {
		outputDir: './'
	},

	cucumberOpts: {
		require: ['./step-definitions.js']
	}
};

step-definitions.js

const { Given, When, Then } = require('@cucumber/cucumber')
const assert = require('assert')

Given('I go on the website {string}', async (url) => {
    await browser.url(url)
})

Then('should the title of the page be {string}', async (expectedTitle) => {
    assert.equal(await browser.getTitle(), expectedTitle)
})

features/my-feature.feature

Feature: Example feature
As a user of Google
I should be able to go to Google and see its correct title

Scenario: Get title of website
Given I go on the website "https://www.google.com/"
Then should the title of the page be "Google"

Now we can run this test on TestingBot!
To start the test, please run this command:

~/node_modules/.bin/wdio run wdio.conf.js

The test will now run on TestingBot. At the end of the test, the testname and success state will be available on TestingBot, together with a video and other meta-data.

Mocha Example

To run Mocha tests on TestingBot, please follow these steps:

npm install @wdio/testingbot-service --save-dev
npm install @wdio/dot-reporter --save-dev
npm install @wdio/mocha-framework --save-dev

Now we'll make a simple Mocha test which uses Firefox on TestingBot to go to Google.com and verify the page's title.

Please create the following files:

wdio.conf.js

exports.config = {

	/**
	 * specify test files
	 */
	specs: [
		'./runner-specs/mocha.test.js'
	],

	/**
	 * capabilities
	 */
	capabilities: [{
		browserName: 'chrome',
		browserVersion: 'latest',
		platformName: 'WIN10'
	}],

	/**
	 * test configurations
	 */
	logLevel: 'silent',
	coloredLogs: true,
	screenshotPath: 'screenshots',
	waitforTimeout: 10000,
	connectionRetryTimeout: 480000,
	framework: 'mocha',
	services: [
        ['testingbot']
    ],
	user: 'api_key',
	key: 'api_secret',

	reporters: ['dot'],
	reporterOptions: {
		outputDir: './'
	},

	mochaOpts: {
		ui: 'bdd'
	}
};

runner-specs/mocha.test.js

const assert = require('assert');
describe('google page', async function() {
	it('should have the right title', async function () {
		await browser.url('https://www.google.com');
		const title = await browser.getTitle();
		assert.equal(title, 'Google');
	});
});

Now we can run this test on TestingBot.
To start the test, please run this command:

~/node_modules/.bin/wdio run wdio.conf.js

The test will now run on TestingBot. At the end of the test, the testname and success state will be available on TestingBot, together with a video and other meta-data.

Jasmine Example

To run Jasmine tests on TestingBot, please follow these steps:

npm install @wdio/testingbot-service --save-dev
npm install @wdio/dot-reporter --save-dev
npm install @wdio/jasmine-framework --save-dev

Now we'll make a simple Jasmine test which uses Firefox on TestingBot to go to Google.com and verify the page's title.

Please create the following files:

wdio.conf.js

exports.config = {

	/**
	 * specify test files
	 */
	specs: [
		'./runner-specs/jasmine.spec.js'
	],

	/**
	 * capabilities
	 */
	capabilities: [{
		browserName: 'chrome',
		browserVersion: 'latest',
		platformName: 'WIN10'
	}],

	/**
	 * test configurations
	 */
	logLevel: 'silent',
	coloredLogs: true,
	screenshotPath: 'screenshots',
	waitforTimeout: 10000,
	connectionRetryTimeout: 480000,
	framework: 'jasmine',
	services: [
        ['testingbot']
    ],
	user: 'api_key',
	key: 'api_secret',

	reporters: ['dot'],
	reporterOptions: {
		outputDir: './'
	},

	jasmineNodeOpts: {
		defaultTimeoutInterval: 9999999
	}
};

runner-specs/jasmine.spec.js

describe('google page', function() {
	it('should have the right title', async function () {
		await browser.url('https://www.google.com');
		const title = await browser.getTitle();
		expect(title).toBe('Google');
	});
});

Now we can run this test on TestingBot.
To start the test, please run this command:

~/node_modules/.bin/wdio run wdio.conf.js

The test will now run on TestingBot. At the end of the test, the testname and success state will be available on TestingBot, together with a video and other meta-data.

Specify Browsers & Devices

To let TestingBot know on which browser/platform/device you want to run your test on, you need to specify the browsername, version, OS and other optional options in the capabilities field.

1. Select a Platform

2. Select a browser

When running mobile tests with WebdriverIO, especially on real devices, make sure to specify a large enough connectionRetryTimeout in case a device is in use:

connectionRetryTimeout: 480000

Testing Internal Websites

We've built TestingBot Tunnel, to provide you with a secure way to run tests against your staged/internal webapps.
Please see our TestingBot Tunnel documentation for more information about this easy to use tunneling solution.

The example below shows how to easily run a WebDriverIO test with our Tunnel:

1. Adjust your wdio.conf.js configuration to use tbTunnel: true.
The wdio-testingbot-service will automatically download and use the tunnel for your tests.

wdio.conf.js

exports.config = {
	services: [
        ['testingbot', {
            tbTunnel: true
        }]
    ],
	user: 'api_key',
	key: 'api_secret'
};

Run Tests in Parallel

Parallel testing allows you to run multiple tests at the same time, significantly reducing your total test execution time.

You can run the same test across different browser configurations, or run different tests on the same configuration — all simultaneously.
TestingBot provides a large grid of browsers and devices, enabling you to run tests in parallel efficiently. This is one of our core features designed to help you test faster and ship more reliably.

With WebDriverIO, you can enable parallel test execution by setting the maxInstances property in your wdio.conf.js file and specifying multiple capabilities. This defines how many instances of the same browser can run in parallel. We recommend setting this value to match the number of parallel sessions allowed by your TestingBot plan.

exports.config = {
	maxInstances: 5, // Adjust this based on your TestingBot plan
	capabilities: [{
		browserName: 'chrome',
		browserVersion: 'latest',
		platformName: 'WIN10'
	}, {
		browserName: 'firefox',
		browserVersion: 'latest',
		platformName: 'WIN10'
	}]
};

Queuing

Every TestingBot plan includes a fixed number of parallel test slots.
If you exceed this limit, additional tests will be automatically queued and executed as soon as slots become available, no tests will be dropped or skipped.

Mark tests as passed/failed

As TestingBot has no way to dermine whether your test passed or failed (it is determined by your business logic), we offer a way to send the test status back to TestingBot. This is useful if you want to see if a test succeeded or failed from the TestingBot member area.

If you are using the wdio-testingbot-service then your tests will automatically report back meta-data to TestingBot (like test success, name, stacktrace, ...)

Common errors

Below are some common errors you might encounter when running WebDriverIO tests on TestingBot:

UND_ERR_HEADERS_TIMEOUT
Problem: This error indicates that WebdriverIO did not receive a response in time. This might happen when requesting a device that is currently in use, or in case you are trying to run more tests in parallel than your plan allows and the tests have been queued for a long time.

Solution: You can try adding the connectionRetryTimeout in your WebDriverIO configuration with a high enough value:
```
connectionRetryTimeout: 480000
```

Other NodeJS Framework examples

WebDriverIO
With WebDriverIO you can run Mocha, Jasmine and Cucumber tests.
CodeceptJS
Run acceptance tests with CodeceptJS on TestingBot.
Protractor
Protractor is an end-to-end test framework for AngularJS applications. Protractor is a NodeJS program built on top of WebDriverJS.
Soda
Selenium Node Adapter. A light-weight Selenium RC client for NodeJS.
Nightwatch
Nightwatch is an automated testing framework written in NodeJS.
WD.js
WD.js is a NodeJS client for WebDriver/Selenium.
Hermione
Hermione is a Test Framework similar to WebDriverIO, with automatic test retries and plugins.