Python Visual UI Testing

Question 1

Why should I use Automated Visual Testing?

Answer

Using Visual Testing as part of your automated test suite is a good idea, for both website based testing as well as mobile app testing.

Here are some of the advantages of using automated UI testing over manual testing:

Improved accuracy and test coverage:
Automation tests are more accurate than manual tests. Automation will consistently check each part of the UI, every single time.
Manual testers might miss visual defects, due to human error, fatigue or any other number of factors.
Improved repeatability:
When integrated in a CI/CD system, you can run these automated UI tests at a consistent interval. For example after every commit or deploy of new code.
Increased speed:
It might take time to set up, configure and program your visual test cases in Python. But once you have these in place, they will prove to be more efficient and faster than going through the UI with a manual test.
Pixel perfect:
Because the UI is compared by an algorithm, it will make sure your UI looks perfect. The automated test will ensure each pixel looks good, by using a visual diff algorithm.

Question 2

Why use Python to do Visual Regression Testing?

Answer

Python is a very popular programming language among many web and mobile app developers. It has grown in popularity in recent years because of its readability and simplicity to create high performance code. Python is cross platform, which means it can be used on Windows, Linux, macOS and other operating systems.

One popular Python test framework is called PyTest, which has plugins that you can install. One PyTest plugin is called pytest-needle which allows you to take and compare screenshots from a remote browser, in a PyTest script.

Question 3

Create your first Python Visual Test

To use Needle with PyTest, you will need to install these packages

pip install selenium pytest pytest-needle

@pytest.mark.element
def test_example_testingbot(needle):
    needle.driver.get('https://testingbot.com')

    # Take an element screen diff
    needle.assert_screenshot('html', (By.Name, 'html'))

You can then run the test on a TestingBot browser, thanks to the built-in support for TestingBot:

pytest --driver TestingBot --capability browserName chrome --capability version latest --capability platform WIN10

To use Needle with Nose, you will need to install these packages

pip install selenium nose needle

from needle.cases import NeedleTestCase

class TestingBotTest(NeedleTestCase):
	@classmethod
    def get_web_driver(cls):
        return NeedleRemote(webdriver.Remote(
		    command_executor='http://key:secret@hub.testingbot.com/wd/hub',
		    desired_capabilities=desired_caps))

    def test_page(self):
        self.driver.get('https://testingbot.com')
        self.assertScreenshot('html', 'html')

You can then run the test with Nose:

nosetests test_testingbot.py --with-save-baseline

Answer

To get started, you can choose to either use PyTest with the needle plugin or use Needle together with Nose.

You can then write your first test script, where you will check if the screenshot you take from TestingBot is exactly the same as your golden image.

Question 4

Best practices when creating visual regression testcases

Answer

Here are some tips to make sure you make the most efficient use of Visual Automated Testing.

Compare components instead of pages:

It is faster and more efficient to compare multiple DOM components, instead of entire viewports/pages.
This is a more modular approach, with clearer error messages. If a test fails, you'll know exactly which component is failing.
Integrate visual testing with other types of testing:

Visual testing should be used in combination with other types of tests, such as functional tests. Depending on pixel diff algorithms alone is not a very good test strategy.

You should make sure you have clear functional tests implemented as well. These test the behaviour of your product, instead of the visual aspect.