Visual Testing

How does one test if the UI being rendered appears correctly to the users or how to test if each UI element appears in the right position and size? The traditional way to test the UI of the application has always been manually, which is time consuming.

Visual testing with help of CodeceptJS will help in improving such use cases for the QA folks.

By default CodeceptJS uses WebDriver helper and Selenium to automate browser. It is also capable of taking screenshots of the application and this could be used for visual testing.

Currently there are two helpers available for Visual testing with CodeceptJS

Using Resemble helper

Resemble.js (opens new window) is a great tool for image comparison and analysis, which can be used with CodeceptJS

Setup

To install the package, just run

npm install codeceptjs-resemblehelper --save

Configuring

This helper should be added to codecept.conf.js config file.

Example:

{
   "helpers": {
     "ResembleHelper" : {
       "require": "codeceptjs-resemblehelper",
       "screenshotFolder" : "./tests/output/",
       "baseFolder": "./tests/screenshots/base/",
       "diffFolder": "./tests/screenshots/diff/"
     }
   }
}

To use the Helper, users must provide the three parameters:

• screenshotFolder : This will always have the same value as output in Codecept configuration, this is the folder where WebDriver saves a screenshot when using I.saveScreenshot method
• baseFolder: This is the folder for base images, which will be used with screenshot for comparison
• diffFolder: This will the folder where resemble would try to store the difference image, which can be viewed later.

Usage

Details about the helper can be found on the Github Repo (opens new window)

Base Image is compared with the screenshot image and test results are derived based on the mismatch tolerance level provided by the user for the comparison

Example

Lets consider visual testing for CodeceptJS Home (opens new window)

Feature('To test screen comparison with resemble Js Example test');

Scenario('Compare CodeceptIO Home Page @visual-test', async ({ I }) => {
    I.amOnPage("/");
    I.saveScreenshot("Codecept_IO_Screenshot_Image.png");
    I.seeVisualDiff("Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false});
});

In this example, we are setting the expected mismatch tolerance level as 2

Base Image (Generated by User)

Screenshot Image (Generated by Test)

Clearly the difference in both the images visible to human eye is the section about Scenario Driven

Diff Image generated by the helper clearly highlights the section which don't match

Failed Test output

To test screen comparison with resemble Js Example test --
  Compare CodeceptIO Home Page @visual-test
    I see Visual Diff "Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false}
MisMatch Percentage Calculated is 2.85
  ✖ FAILED in 418ms


-- FAILURES:

  1) To test screen comparison with resemble Js Example test
       Compare CodeceptIO Home Page @visual-test:

      MissMatch Percentage 2.85
      + expected - actual

      -false
      +true

Codeceptjs-resemblehelper basically comes with two major functions

1. seeVisualDiff which can be used to compare two images and calculate the misMatch percentage.
2. seeVisualDiffForElement which can be used to compare elements on the two images and calculate misMatch percentage.

Using Applitools

Applitools helps Test Automation engineers, DevOps, and FrontEnd Developers continuously test and validate visually perfect mobile, web, and native apps. Now it can be used with CodeceptJS.

Setup

Create an account at Applitools (opens new window) and install the npm packages

npm i codeceptjs-applitoolshelper --save
npm i webdriverio@5 --save

Configuring

...
  helpers: {
    WebDriver: {
      url: 'https://applitools.com/helloworld',
      browser: 'chrome',
      desiredCapabilities: {
        chromeOptions: {
          args: [ '--headless', '--disable-extensions', '--disable-gpu', '--no-sandbox', '--disable-dev-shm-usage']
        }
      },
      windowSize: '1920x600',
      smartWait: 5000,
      timeouts: {
        'script': 60000,
        'page load': 10000
      },
    },
    ApplitoolsHelper: {
      require: 'codeceptjs-applitoolshelper',
      applitoolsKey: 'YOUR_API_KEY'
    }
  },
...

To use this helper you need to provide the following info:

• applitoolsKey (Required): You can find your API key under the user menu located at the right hand side of the test manager toolbar
• windowSize (Optional): the windows size as for instance 1440x700, if not provided, the default 1920x600 will be used. The windowSize will follow this precedence: ApplitoolsHelper, Webdriver.
• appName (Optional): you can either provide your desired application name, if not provided, the default 'Application Under Test' will be used.

Usage

/**
 * @param pageToCheck {string} Name of page you want to check
 */
I.eyeCheck(pageToCheck);

The first time you run this test a new baseline will be created, and subsequent test runs will be compared to this baseline. If any screenshot mismatch its baseline image in a perceptible way, there will be a DiffsFoundException which includes a URL that points to a detailed report where you can see the detected differences and take appropriate actions such as reporting bugs, updating the baseline and more.

-- FAILURES:

  1) Applitools functionality
       Check home page @test:
     Test 'Applitools functionality' of 'Application Under Test' detected differences!. See details at: https://eyes.applitools.com/app/batches/00000251831777088983/00000251831777088717?accountId=KO-Oh9tXI0e8VF8Ha_GLVA~~

You can find the latest documentation here Applitools Docs (opens new window)

Example

Lets consider visual testing for CodeceptJS Home (opens new window). You can also find example repo here: https://github.com/PeterNgTr/codeceptjs-applitoolshelper

const { I } = inject();

Feature('Applitools functionality');

Before(() => {
    I.amOnPage('https://applitools.com/helloworld');
});

Scenario('Check home page @test', async ({  }) => {
    await I.eyeCheck('Homepage');
});