Features

Python Visual UI Testing

  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share on HackerNews

Visual regression testing is a specific type of regression testing, which verifies the aesthetic accuracy of what end-users of the product see.

When you deploy new code, you'll want to make sure your website still looks the same. Most of the time, you would compare two screenshots.

The first screenshot should have been created as a baseline. A baseline is a state of what you want your UI to look like.

Once you run the regression test, you will take another screenshot of the same UI and detect if there are any visual differences.

By using a combination of Python tests and the power of TestingBot's browser and device grid, you can make sure that your UI looks perfect across a large combination of end user devices.

Why should I use Automated Visual Testing?

Using Visual Testing as part of your automated test suite is a good idea, for both website based testing as well as mobile app testing.

Here are some of the advantages of using automated UI testing over manual testing:

  • Improved accuracy and test coverage:

    Automation tests are more accurate than manual tests. Automation will consistently check each part of the UI, every single time.
    Manual testers might miss visual defects, due to human error, fatigue or any other number of factors.

  • Improved repeatability:

    When integrated in a CI/CD system, you can run these automated UI tests at a consistent interval. For example after every commit or deploy of new code.

  • Increased speed:

    It might take time to set up, configure and program your visual test cases in Python. But once you have these in place, they will prove to be more efficient and faster than going through the UI with a manual test.

  • Pixel perfect:

    Because the UI is compared by an algorithm, it will make sure your UI looks perfect. The automated test will ensure each pixel looks good, by using a visual diff algorithm.

Why use Python to do Visual Regression Testing?

Python is a very popular programming language among many web and mobile app developers. It has grown in popularity in recent years because of its readability and simplicity to create high performance code. Python is cross platform, which means it can be used on Windows, Linux, macOS and other operating systems.

One popular Python test framework is called PyTest, which has plugins that you can install. One PyTest plugin is called pytest-needle which allows you to take and compare screenshots from a remote browser, in a PyTest script.

You can use a Selenium, Playwright or Puppeteer driver together with pytest-needle to take a screenshot. The plugin will then use Needle's pixel comparison to go over all pixels of both the base image and the screenshot to detect any differences.

Create your first Python Visual Test

To get started, you can choose to either use PyTest with the needle plugin or use Needle together with Nose.

You can then write your first test script, where you will check if the screenshot you take from TestingBot is exactly the same as your golden image.

To use Needle with PyTest, you will need to install these packages

Copy code
pip install selenium pytest pytest-needle
Copy code
@pytest.mark.element
def test_example_testingbot(needle):
    needle.driver.get('https://testingbot.com')

    # Take an element screen diff
    needle.assert_screenshot('html', (By.Name, 'html'))

You can then run the test on a TestingBot browser, thanks to the built-in support for TestingBot:

Copy code
pytest --driver TestingBot --capability browserName chrome --capability version latest --capability platform WIN10

To use Needle with Nose, you will need to install these packages

Copy code
pip install selenium nose needle
Copy code
from needle.cases import NeedleTestCase

class TestingBotTest(NeedleTestCase):
	@classmethod
    def get_web_driver(cls):
        return NeedleRemote(webdriver.Remote(
		    command_executor='http://key:secret@hub.testingbot.com/wd/hub',
		    desired_capabilities=desired_caps))

    def test_page(self):
        self.driver.get('https://testingbot.com')
        self.assertScreenshot('html', 'html')

You can then run the test with Nose:

Copy code
nosetests test_testingbot.py --with-save-baseline

Best practices when creating visual regression testcases

Here are some tips to make sure you make the most efficient use of Visual Automated Testing.

  1. Compare components instead of pages:

    It is faster and more efficient to compare multiple DOM components, instead of entire viewports/pages.
    This is a more modular approach, with clearer error messages. If a test fails, you'll know exactly which component is failing.

  2. Integrate visual testing with other types of testing:

    Visual testing should be used in combination with other types of tests, such as functional tests. Depending on pixel diff algorithms alone is not a very good test strategy.

    You should make sure you have clear functional tests implemented as well. These test the behaviour of your product, instead of the visual aspect.

  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
  • Share on HackerNews
TestingBot Logo

Sign up for a Free Trial

Start testing your apps with TestingBot.

No credit card required.

Other Articles

Android Espresso Tutorial

Learn more about Android Espresso Testing in this tutorial.

Read more
Page Object Model (POM) and Page Factory with Appium

Learn more about the Page Object Model and how to use it with Appium.

Read more
Getting Started with Appium

Learn how to use Appium for automated testing. We will provide some tips and tricks, performance optimizations and ways to use Appium Inspector to troubleshoot your native mobile app testing.

Read more
Getting Started with IntelliJ and Selenium WebDriver

A brief overview on how to setup, configure and run Selenium WebDriver tests with IntelliJ IDEA.

Read more
Internationalization testing of websites and mobile apps

Learn more about Internationalization Testing: how to prepare and test your website or mobile app for different regions in the world.

Read more
Visual Testing with Playwright

Playwright provides automated browser testing. It offers a built-in feature to perform visual regression testing for your website.

Read more