Create and understand Automated Tests with ChatGPT

ChatGPT is a popular language model developed by OpenAI. It is designed to facilitate natural and engaging conversations with users and respond to any prompt that you provide. It uses the power of the GPT-3.5 (and GPT-4) architecture. The engine has been trained on a diverse range of internet text, making it highly proficient in understanding and generating human-like language.

There are several advantages in using ChatGPT (or an other language model), to help writing tests or generate tests from scratch:

• Code generation: You can use AI to generate entire pieces of code by asking for it in a prompt. Most of the time, the AI engine will return code snippets which you can use to create test scripts.
• Code completion: If you already have existing Selenium or Appium tests, you can feed this to ChatGPT and ask for help with code that you are stuck with.
• Code explanation: Let's assume you've inherited a code-base of test scripts and you're not sure what the code does exactly. You can ask ChatGPT for an explanation of the code.
• Code review: You can even ask ChatGPT to review some of the Puppeteer/Playwright code you've written. It will be able to detect errors/bugs, or recommend code to make improve your tests.

Obviously, as ChatGPT is an automated tool, we do recommend verifying all code that is generated by it. AI is a great way to automate things, but human verification is still required.

Generating Selenium tests with ChatGPT requires using the language model to help in creating test automation scripts. While using an AI tool can provide guidance and suggestions, it's important to remember that it doesn't have previous knowledge of your tests or product that you are testing. Its responses might not always be perfect, but they can help in speeding up creating new Selenium tests.

Below are some tips on how to generate Selenium tests:

• Context Setup: Familiarize ChatGPT with your project and its test requirements. Provide information about the website or application you're testing. Don't forget to provide the specific test scenarios you want to cover.
• Generate a prompt: Good prompts are very important when using a generative AI tool. One example could be: I need assistance in generating a Selenium test script, written in NodeJS, to verify the login functionality of a web application that is located on this page: ....
• Describe a Test Scenario: Clearly outline the test scenario or user interaction you want to automate. Provide details such as the steps, expected outcomes, and any specific elements or conditions you need to validate. Make sure you include as much details as possible.
• Iterate and Refine: Review all code suggestions, as ChatGPT might not always produce accurate or optimal code. If you spot an issue, you can modify the code snippet or provide feedback to ChatGPT.
• Feedback Loop: Provide feedback to ChatGPT on the quality and relevance of the generated code. This helps improve the model's future responses and ensures better assistance in generating Selenium tests.

Remember that using an AI generator can be a helpful tool in generating Selenium tests. It is not a replacement for understanding the Selenium framework, programming concepts or software testing best-practices. We recommended you have a solid foundation in these areas to produce effective and reliable test scripts, before attempting to resort to using an AI generative text generator.

You can use the tips below to help with generating Appium tests using ChatGPT. This will allow you to generate Appium tests more quickly. Be sure to validate all code snippets generate from the Text generation AI though, as it is not fault-proof.

• Familiarize AI: Introduce ChatGPT to your project context. Provide information about the mobile app you want to test. Which category does this mobile app belong to, what are you looking to test, would you like to test on Android or iOS, etc.
• Scenario Description: Clearly describe the specific test scenario or user interaction you want to automate. Outline the steps, expected outcomes, and any particular elements or conditions that need validation.
• Programming Language: Tell ChatGPT which programming language you want to use, as Appium provides libraries for several programming languages, including NodeJS, C#, Ruby and more.
• Feedback Loop: Provide feedback to AI language model on the quality and relevance of the generated Appium code. This will help the language model in improving its results, in future prompts, even in the current session.

Be cautious not to simply copy and paste the generated code, as it might contain bugs. Appium tests often require knowledge about mobile app UI elements, selectors, and platform-specific considerations, which should be considered while refining the generated code.

Puppeteer is an automation framework, designed to automate Chromium based browsers. Currently it is not possible to automate mobile apps or other browsers. You can use ChatGPT to write Puppeteer scripts for you. Please see the example prompts below on how to achieve this.

Puppeteer ChatGPT Prompt

The generated code should be similar to the following:

Make sure to install Puppeteer by running npm install puppeteer before running the script. This script launches a browser instance, navigates to the TestingBot website, validates the title, clicks the 'Sign up' button, fills in the form, clicks the sign-up button on the form, and finally validates the confirmation page URL. Adjust the selectors and input values according to the website's structure and your testing needs.

Playwright is an automation framework, created by Microsoft, which is capable of automating Chromium based browsers, Firefox and WebKit. In the section below, we will show you an example prompt on how to generate a code snippet that will run a Playwright test against a website.

TestingBot.com is a website. Prepare a Playwright test to carry out the following validations on the website. The script should run on Firefox.

Open testingbot.com. Success if page loads with the following word included in the title 'TestingBot'
Click the 'Login up' button and fill in the form with the necessary information.
Once the form is filled in, click the log in button.
Success if page redirects to the member dashboard page.
				

const { firefox } = require('playwright');

(async () => {
  const browser = await firefox.launch();
  const context = await browser.newContext();

  const page = await context.newPage();
  try {
    // Open testingbot.com

    await page.goto('https://testingbot.com');
    
    // Validate title contains 'TestingBot'

    const pageTitle = await page.title();
    if (pageTitle.includes('TestingBot')) {
      console.log('Page loaded with the expected title: TestingBot');
    } else {
      console.error('Page title does not contain the expected word: TestingBot');
    }
    
    // Click the 'Login' button

    await page.click('a:has-text("Login")');
    
    // Fill in the login form

    await page.fill('input[name="email"]', 'your@email.com');
    await page.fill('input[name="password"]', 'yourpassword');
    
    // Click the login button

    await page.click('button:has-text("Log in")');
    
    // Wait for navigation to the member dashboard page

    await page.waitForNavigation();
    
    // Validate the member dashboard page URL

    if (page.url().includes('/members')) {
      console.log('Successfully redirected to the member dashboard page');
    } else {
      console.error('Did not redirect to the expected member dashboard page');
    }
  } catch (error) {
    console.error('An error occurred:', error);
  } finally {
    await context.close();
    await browser.close();
  }
})();

You can use ChatGPT to interpret test scripts; to understand what they do. For example, let's say you have this piece of Appium script and you would like an explanation of what it does:

You can ask ChatGPT what this code does, with the following prompt: Please explain what the following Appium code does

ChatGPT will return a lengthy explanation of what the test script does. It will mention the import of dependencies at the top of the script, the credentials needed to connect to the TestingBot grid, the capabilities that are used to run the Appium test on a specific device and further explain what all the steps do.

Data analysis and summarisation

ChatGPT can interpret test results as well. Let's say for example that you've received a JUnit XML report and you would like to know what it means:

By providing a prompt to ChatGPT, it will explain the results: Please explain the following JUnit Report XML file.

ChatGPT will break down the result file in several sections, each time explaining what happened. It will explain what the testcase annotation means, how many tests failed and which tests passed.

Using ChatGPT to generate test cases comes with its own set of pros and cons. Here's an overview:

Advantages of using ChatGPT

• Rapid Idea Generation: ChatGPT can quickly provide ideas for test cases, especially for complex scenarios or edge cases that might be overlooked.
• Variety of Scenarios: ChatGPT can help generate a wide range of test cases, covering different aspects of the application, improving test coverage. It might come up with things you would not have considered.
• Assistance in designing tests: It can assist in formulating test scenarios and designing test cases that are well-structured and cover relevant functionalities.
• Quick Prototyping: ChatGPT can be useful for quickly prototyping test cases during initial development phases or for exploratory testing. You can then refine the test scenarios manually to your liking.

Disadvantages of using ChatGPT

• Lack of Domain Knowledge: ChatGPT might lack specific domain knowledge or context about the application being tested, leading to inaccurate or irrelevant test case suggestions.
• Quality of Tests: The generated test cases might not follow established testing standards, leading to inconsistent or insufficient test coverage.
• Model Limitations: ChatGPT might sometimes produce inaccurate suggestions, which could negatively impact the quality of generated test cases.