Create and understand Automated Tests with ChatGPT

ChatGPT is a popular language model developed by OpenAI. It is designed to facilitate natural and engaging conversations with users and respond to any prompt that you provide. It uses the power of the GPT-3.5 (and GPT-4) architecture. The engine has been trained on a diverse range of internet text, making it highly proficient in understanding and generating human-like language.

There are several advantages in using ChatGPT (or an other language model), to help writing tests or generate tests from scratch:

• Code generation: You can use AI to generate entire pieces of code by asking for it in a prompt. Most of the time, the AI engine will return code snippets which you can use to create test scripts.
• Code completion: If you already have existing Selenium or Appium tests, you can feed this to ChatGPT and ask for help with code that you are stuck with.
• Code explanation: Let's assume you've inherited a code-base of test scripts and you're not sure what the code does exactly. You can ask ChatGPT for an explanation of the code.
• Code review: You can even ask ChatGPT to review some of the Puppeteer/Playwright code you've written. It will be able to detect errors/bugs, or recommend code to make improve your tests.

Obviously, as ChatGPT is an automated tool, we do recommend verifying all code that is generated by it. AI is a great way to automate things, but human verification is still required.

Generating Selenium tests with ChatGPT requires using the language model to help in creating test automation scripts. While using an AI tool can provide guidance and suggestions, it's important to remember that it doesn't have previous knowledge of your tests or product that you are testing. Its responses might not always be perfect, but they can help in speeding up creating new Selenium tests.

Below are some tips on how to generate Selenium tests:

• Context Setup: Familiarize ChatGPT with your project and its test requirements. Provide information about the website or application you're testing. Don't forget to provide the specific test scenarios you want to cover.
• Generate a prompt: Good prompts are very important when using a generative AI tool. One example could be: I need assistance in generating a Selenium test script, written in NodeJS, to verify the login functionality of a web application that is located on this page: ....
• Describe a Test Scenario: Clearly outline the test scenario or user interaction you want to automate. Provide details such as the steps, expected outcomes, and any specific elements or conditions you need to validate. Make sure you include as much details as possible.
• Iterate and Refine: Review all code suggestions, as ChatGPT might not always produce accurate or optimal code. If you spot an issue, you can modify the code snippet or provide feedback to ChatGPT.
• Feedback Loop: Provide feedback to ChatGPT on the quality and relevance of the generated code. This helps improve the model's future responses and ensures better assistance in generating Selenium tests.

Remember that using an AI generator can be a helpful tool in generating Selenium tests. It is not a replacement for understanding the Selenium framework, programming concepts or software testing best-practices. We recommended you have a solid foundation in these areas to produce effective and reliable test scripts, before attempting to resort to using an AI generative text generator.

You can use the tips below to help with generating Appium tests using ChatGPT. This will allow you to generate Appium tests more quickly. Be sure to validate all code snippets generate from the Text generation AI though, as it is not fault-proof.

• Familiarize AI: Introduce ChatGPT to your project context. Provide information about the mobile app you want to test. Which category does this mobile app belong to, what are you looking to test, would you like to test on Android or iOS, etc.
• Scenario Description: Clearly describe the specific test scenario or user interaction you want to automate. Outline the steps, expected outcomes, and any particular elements or conditions that need validation.
• Programming Language: Tell ChatGPT which programming language you want to use, as Appium provides libraries for several programming languages, including NodeJS, C#, Ruby and more.
• Feedback Loop: Provide feedback to AI language model on the quality and relevance of the generated Appium code. This will help the language model in improving its results, in future prompts, even in the current session.

Be cautious not to simply copy and paste the generated code, as it might contain bugs. Appium tests often require knowledge about mobile app UI elements, selectors, and platform-specific considerations, which should be considered while refining the generated code.

Puppeteer is an automation framework, designed to automate Chromium based browsers. Currently it is not possible to automate mobile apps or other browsers. You can use ChatGPT to write Puppeteer scripts for you. Please see the example prompts below on how to achieve this.

Puppeteer ChatGPT Prompt

TestingBot.com is a website. Prepare a Puppeteer test to carry out the following validations on the website:

Open testingbot.com. Success if page loads with the following word included in the title 'TestingBot'
Click the 'Sign up' button and fill in the form with the necessary information.
Once the form is filled in, click the sign up button.
Success if page redirects to a confirmation page.
				

The generated code should be similar to the following:

const puppeteer = require('puppeteer');

(async () => {
  // Launch a new browser instance

  const browser = await puppeteer.launch();
  
  // Create a new page

  const page = await browser.newPage();
  
  try {
    // Open testingbot.com

    await page.goto('https://testingbot.com');
    
    // Validate title contains 'TestingBot'

    const pageTitle = await page.title();
    if (pageTitle.includes('TestingBot')) {
      console.log('Page loaded with the expected title: TestingBot');
    } else {
      console.error('Page title does not contain the expected word: TestingBot');
    }
    
    // Click the 'Sign up' button

    await page.click('a[href="/auth/register"]');
    
    // Fill in the sign-up form

    await page.type('#name', 'Your Name');
    await page.type('#email', 'your@email.com');
    await page.type('#password', 'yourpassword');
    
    // Click the sign-up button

    await page.click('button[type="submit"]');
    
    // Wait for navigation to the confirmation page

    await page.waitForNavigation();
    
    // Validate the confirmation page URL

    if (page.url().includes('/auth/register/confirm')) {
      console.log('Successfully redirected to the confirmation page');
    } else {
      console.error('Did not redirect to the expected confirmation page');
    }
  } catch (error) {
    console.error('An error occurred:', error);
  } finally {
    // Close the browser

    await browser.close();
  }
})();

Make sure to install Puppeteer by running npm install puppeteer before running the script. This script launches a browser instance, navigates to the TestingBot website, validates the title, clicks the 'Sign up' button, fills in the form, clicks the sign-up button on the form, and finally validates the confirmation page URL. Adjust the selectors and input values according to the website's structure and your testing needs.

Playwright is an automation framework, created by Microsoft, which is capable of automating Chromium based browsers, Firefox and WebKit. In the section below, we will show you an example prompt on how to generate a code snippet that will run a Playwright test against a website.

You can use ChatGPT to interpret test scripts; to understand what they do. For example, let's say you have this piece of Appium script and you would like an explanation of what it does:

var wd = require('wd');
var assert = require('assert');
var asserters = wd.asserters;
var testingbotKey = "api_key",
testingbotSecret = "api_secret";

desiredCaps = {
  'name': 'My First App Test',
  'deviceName' : 'Galaxy S8',
  'platformName': 'Android',
  'version': '9.0',
  'app' : 'https://testingbot.com/appium/sample.apk',
  'realDevice': true
};
driver = wd.promiseRemote("http://" + testingbotKey + ":" + testingbotSecret + "@hub.testingbot.com/wd/hub");

driver
  .init(desiredCaps)
  .then(function () {
    return driver.waitForElementByAccessibilityId('inputA', asserters.isDisplayed && asserters.isEnabled, 30000);
  })
  .then(function (inputA) {
    return inputA.sendKeys("10");
  })
  .then(function () {
    return driver.waitForElementByAccessibilityId('inputB', asserters.isDisplayed && asserters.isEnabled, 30000);
  })
  .then(function (inputB) {
    return inputB.sendKeys("5");
  })
  .then(function () {
    return driver.waitForElementByAccessibilityId('sum', asserters.isDisplayed && asserters.isEnabled, 30000);
  })
  .then(function (textOutput) {
    return textOutput.text().then(function(value) {
      if (value === "15")
        assert(true);
      else
        assert(false);
    });
  })
  .fin(function() { return driver.quit(); })
  .done();

You can ask ChatGPT what this code does, with the following prompt: Please explain what the following Appium code does

ChatGPT will return a lengthy explanation of what the test script does. It will mention the import of dependencies at the top of the script, the credentials needed to connect to the TestingBot grid, the capabilities that are used to run the Appium test on a specific device and further explain what all the steps do.

Data analysis and summarisation

ChatGPT can interpret test results as well. Let's say for example that you've received a JUnit XML report and you would like to know what it means:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="Playwright Test Suite" tests="3" failures="0" errors="0" skipped="0" timestamp="2023-08-12T12:10:00Z" time="12.345">
<testcase classname="Homepage Test" name="Verify homepage loads correctly" time="3.456"></testcase>
<testcase classname="Homepage Test" name="Verify title is correct" time="2.345"></testcase>
<testcase classname="Homepage Test" name="Verify navigation links" time="6.789" failures="1">
<failure message="Navigation link 'Pricing' not found">
Timed out retrying: expected to find element: '.nav-link[href="/pricing"]', but never found it.
</failure>
</testcase>
</testsuite>
</testsuites>

By providing a prompt to ChatGPT, it will explain the results: Please explain the following JUnit Report XML file.

ChatGPT will break down the result file in several sections, each time explaining what happened. It will explain what the testcase annotation means, how many tests failed and which tests passed.