Electron App Testing with Playwright

Prepare your Electron app

TestingBot needs to be able to download your Electron app as a zip file. You can host this zip file yourself, for example on AWS S3 or GitHub Releases, or upload it to TestingBot Storage, which returns a tb:// URL you can use instead.

Next to the URL of the zip file, you also need to know the path of the actual Electron binary inside the zip file. This is the binary_location parameter, so TestingBot knows which file to launch. For the demo app on macOS this is:

testingbot-electron-demo-app.app/Contents/MacOS/testingbot-electron-demo-app

On Windows, it is simply the executable in the zip file:

testingbot-electron-demo-app.exe

See App management on the WebDriver Electron page for more details about uploading your app.

Run your first test

Install playwright-core, connect with chromium.connectOverCDP() and pass the Electron parameters in the WebSocket URL. Once connected, your Electron app's window is available as the first page of the first browser context.

The browserVersion parameter must be the major Electron version your app was built with. The demo app was built with Electron 31, so the examples below use browserVersion=31.

npm i playwright-core

const { chromium } = require('playwright-core');

const app = 'https://github.com/testingbot/testingbot-electron-demo-app/releases/download/v1.0.0/testingbot-electron-demo-app-darwin-arm64-1.0.0.zip';
const binaryLocation = 'testingbot-electron-demo-app.app/Contents/MacOS/testingbot-electron-demo-app';

const wsEndpoint = 'wss://cloud.testingbot.com'
  + '?key=api_key&secret=api_secret'
  + '&browserName=electron'
  + '&browserVersion=31'
  + '&platform=TAHOE'
  + `&app=${encodeURIComponent(app)}`
  + `&binary_location=${encodeURIComponent(binaryLocation)}`
  + `&name=${encodeURIComponent('My First Electron Test')}`;

(async () => {
  // Starting the remote machine and downloading your app can take a while
  const browser = await chromium.connectOverCDP(wsEndpoint, { timeout: 300000 });

  const context = browser.contexts()[0];
  const page = context.pages()[0]; // the Electron app window

  // Calculate 3 + 4 in the demo calculator app
  await page.click('#btn-3');
  await page.click('#btn-plus');
  await page.click('#btn-4');
  await page.click('#btn-equal');

  const result = await page.textContent('#calc-display');
  console.log(`3 + 4 = ${result}`);

  await browser.close();
})();

const { chromium } = require('playwright-core');

const app = 'https://github.com/testingbot/testingbot-electron-demo-app/releases/download/v1.0.0/testingbot-electron-demo-app-win32-x64-1.0.0.zip';
const binaryLocation = 'testingbot-electron-demo-app.exe';

const wsEndpoint = 'wss://cloud.testingbot.com'
  + '?key=api_key&secret=api_secret'
  + '&browserName=electron'
  + '&browserVersion=31'
  + '&platform=WIN10'
  + `&app=${encodeURIComponent(app)}`
  + `&binary_location=${encodeURIComponent(binaryLocation)}`
  + `&name=${encodeURIComponent('My First Electron Test')}`;

(async () => {
  // Starting the remote machine and downloading your app can take a while
  const browser = await chromium.connectOverCDP(wsEndpoint, { timeout: 300000 });

  const context = browser.contexts()[0];
  const page = context.pages()[0]; // the Electron app window

  // Calculate 3 + 4 in the demo calculator app
  await page.click('#btn-3');
  await page.click('#btn-plus');
  await page.click('#btn-4');
  await page.click('#btn-equal');

  const result = await page.textContent('#calc-display');
  console.log(`3 + 4 = ${result}`);

  await browser.close();
})();

Playwright Test

You can use the same connection with Playwright Test, Playwright's own test runner. Connect in a beforeAll hook or fixture and run your suite with npx playwright test.

const { test, expect, chromium } = require('@playwright/test');

const app = 'https://github.com/testingbot/testingbot-electron-demo-app/releases/download/v1.0.0/testingbot-electron-demo-app-darwin-arm64-1.0.0.zip';
const binaryLocation = 'testingbot-electron-demo-app.app/Contents/MacOS/testingbot-electron-demo-app';

const wsEndpoint = 'wss://cloud.testingbot.com'
  + '?key=api_key&secret=api_secret'
  + '&browserName=electron'
  + '&browserVersion=31'
  + '&platform=TAHOE'
  + `&app=${encodeURIComponent(app)}`
  + `&binary_location=${encodeURIComponent(binaryLocation)}`
  + `&name=${encodeURIComponent('Electron Playwright Test')}`;

test('calculator adds two numbers', async () => {
  test.setTimeout(600000); // remote machine startup can take a few minutes
  const browser = await chromium.connectOverCDP(wsEndpoint, { timeout: 300000 });

  const page = browser.contexts()[0].pages()[0];

  await page.click('#btn-3');
  await page.click('#btn-plus');
  await page.click('#btn-4');
  await page.click('#btn-equal');

  await expect(page.locator('#calc-display')).toHaveText('7');

  await browser.close();
});

Python

Playwright for Python offers the same connect_over_cdp() API, so you can test your Electron app from Python as well.

pip install playwright

from urllib.parse import quote
from playwright.sync_api import sync_playwright

app = "https://github.com/testingbot/testingbot-electron-demo-app/releases/download/v1.0.0/testingbot-electron-demo-app-darwin-arm64-1.0.0.zip"
binary_location = "testingbot-electron-demo-app.app/Contents/MacOS/testingbot-electron-demo-app"

ws_endpoint = (
    "wss://cloud.testingbot.com"
    "?key=api_key&secret=api_secret"
    "&browserName=electron"
    "&browserVersion=31"
    "&platform=TAHOE"
    f"&app={quote(app, safe='')}"
    f"&binary_location={quote(binary_location, safe='')}"
    "&name=Electron%20Playwright%20Python"
)

with sync_playwright() as p:
    # Starting the remote machine and downloading your app can take a while
    browser = p.chromium.connect_over_cdp(ws_endpoint, timeout=300000)

    page = browser.contexts[0].pages[0]  # the Electron app window

    page.click("#btn-3")
    page.click("#btn-plus")
    page.click("#btn-4")
    page.click("#btn-equal")

    result = page.text_content("#calc-display")
    print(f"3 + 4 = {result}")

    browser.close()

Connection parameters

All parameters are passed as query parameters in the WebSocket URL. Make sure to URL-encode the values, especially the app URL and binary_location path.

Mark tests as passed or failed

Playwright does not report test success or failure to TestingBot by itself. You can use the testingbot_executor custom command from within your test to mark the session as passed or failed in the TestingBot dashboard.

try {
  await expect(page.locator('#calc-display')).toHaveText('7');
  // Mark the test as passed on TestingBot
  await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({
    action: 'setSessionStatus',
    arguments: { passed: true, reason: 'Calculator sum matched' }
  })}`);
} catch (err) {
  // Mark the test as failed on TestingBot
  await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({
    action: 'setSessionStatus',
    arguments: { passed: false, reason: 'Calculator sum did not match' }
  })}`);
  throw err;
}

Supported Playwright features

Playwright talks to your Electron app through the Chrome DevTools Protocol, which supports everything that operates on browser contexts and pages:

• Locators, clicking, typing and all other page interactions
• Assertions with expect()
• Screenshots
• Evaluating JavaScript in the renderer process with page.evaluate()
• Network inspection of requests made by your app's windows

Because your app runs on a remote machine, the Electron-specific APIs from Playwright's local _electron.launch() mode are not available. This means no electronApp.evaluate() access to the Electron main process, such as app or BrowserWindow modules. If you need to assert on main process behavior, expose that state to the renderer process, for example via IPC, and read it with page.evaluate().

Test Results

Every Electron Playwright test appears in the TestingBot member dashboard. For each test you get access to a video recording, log files and additional meta-data.

These results are also available through the TestingBot REST-API or by using one of the TestingBot CI/CD plugins.