Visual regression testing is a specific type of regression testing, which verifies the aesthetic accuracy of what end-users of the product see.
When you deploy new code, you'll want to make sure your website still looks the same. Most of the time, you would compare two screenshots.
The first screenshot should have been created as a baseline. A baseline is a state of what you want your UI to look like.
Once you run the regression test, you will take another screenshot of the same UI and detect if there are any visual differences.
By using a combination of Python tests and the power of TestingBot's browser and device grid, you can make sure that your UI looks perfect across a large combination of end user devices.
Why should I use Automated Visual Testing?
Using Visual Testing as part of your automated test suite is a good idea, for both website based testing as well as mobile app testing.
Here are some of the advantages of using automated UI testing over manual testing:
-
Improved accuracy and test coverage:
Automation tests are more accurate than manual tests. Automation will consistently check each part of the UI, every single time.
Manual testers might miss visual defects, due to human error, fatigue or any other number of factors. -
Improved repeatability:
When integrated in a CI/CD system, you can run these automated UI tests at a consistent interval. For example after every commit or deploy of new code.
-
Increased speed:
It might take time to set up, configure and program your visual test cases in Python. But once you have these in place, they will prove to be more efficient and faster than going through the UI with a manual test.
-
Pixel perfect:
Because the UI is compared by an algorithm, it will make sure your UI looks perfect. The automated test will ensure each pixel looks good, by using a visual diff algorithm.
Why use Python to do Visual Regression Testing?
Python is a very popular programming language among many web and mobile app developers. It has grown in popularity in recent years because of its readability and simplicity to create high performance code. Python is cross platform, which means it can be used on Windows, Linux, macOS and other operating systems.
One popular Python test framework is called PyTest, which has plugins that you can install. One PyTest plugin is called pytest-needle which allows you to take and compare screenshots from a remote browser, in a PyTest script.
You can use a Selenium, Playwright or Puppeteer driver together with pytest-needle to take a screenshot. The plugin will then use Needle's pixel comparison to go over all pixels of both the base image and the screenshot to detect any differences.
Create your first Python Visual Test
To get started, you can choose to either use PyTest with the needle plugin or use Needle together with Nose.
You can then write your first test script, where you will check if the screenshot you take from TestingBot is exactly the same as your golden image.
To use Needle with PyTest, you will need to install these packages
pip install selenium pytest pytest-needle@pytest.mark.element
def test_example_testingbot(needle):
needle.driver.get('https://testingbot.com')
# Take an element screen diff
needle.assert_screenshot('html', (By.Name, 'html'))You can then run the test on a TestingBot browser, thanks to the built-in support for TestingBot:
pytest --driver TestingBot --capability browserName chrome --capability version latest --capability platform WIN10To use Needle with Nose, you will need to install these packages
pip install selenium nose needlefrom needle.cases import NeedleTestCase
class TestingBotTest(NeedleTestCase):
@classmethod
def get_web_driver(cls):
return NeedleRemote(webdriver.Remote(
command_executor='http://key:secret@hub.testingbot.com/wd/hub',
desired_capabilities=desired_caps))
def test_page(self):
self.driver.get('https://testingbot.com')
self.assertScreenshot('html', 'html')You can then run the test with Nose:
nosetests test_testingbot.py --with-save-baseline
