Visual Testing

# Visual Testing

How does one test if the UI being rendered appears correctly to the users or how to test if each UI element appears in the right position and size? The traditional way to test the UI of the application has always been manually, which is time consuming.

Visual testing with help of CodeceptJS will help in improving such use cases for the QA folks.

By default CodeceptJS uses WebDriver helper and Selenium to automate browser. It is also capable of taking screenshots of the application and this could be used for visual testing.

Currently there are two helpers available for Visual testing with CodeceptJS

# Using Resemble helper

Resemble.js is a great tool for image comparison and analysis, which can be used with CodeceptJS

# Setup

To install the package, just run

npm install codeceptjs-resemblehelper --save

# Configuring

This helper should be added to codecept.conf.js config file.

Example:

{
   "helpers": {
     "ResembleHelper" : {
       "require": "codeceptjs-resemblehelper",
       "screenshotFolder" : "./tests/output/",
       "baseFolder": "./tests/screenshots/base/",
       "diffFolder": "./tests/screenshots/diff/"
     }
   }
}

To use the Helper, users must provide the three parameters:

  • screenshotFolder : This will always have the same value as output in Codecept configuration, this is the folder where webdriverIO saves a screenshot when using I.saveScreenshot method
  • baseFolder: This is the folder for base images, which will be used with screenshot for comparison
  • diffFolder: This will the folder where resemble would try to store the difference image, which can be viewed later.

# Usage

Details about the helper can be found on the Github Repo

Base Image is compared with the screenshot image and test results are derived based on the mismatch tolerance level provided by the user for the comparison

# Example

Lets consider visual testing for CodeceptJS Home

Feature('To test screen comparison with resemble Js Example test');

Scenario('Compare CodeceptIO Home Page @visual-test', async (I) => {
    I.amOnPage("/");
    I.saveScreenshot("Codecept_IO_Screenshot_Image.png");
    I.seeVisualDiff("Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false});
});

In this example, we are setting the expected mismatch tolerance level as 2

Base Image (Generated by User) Base Image

Screenshot Image (Generated by Test) Screenshot Image

Clearly the difference in both the images visible to human eye is the section about Scenario Driven

Difference Image

Diff Image generated by the helper clearly highlights the section which don't match

Highlight

Failed Test output

To test screen comparison with resemble Js Example test --
  Compare CodeceptIO Home Page @visual-test
    I see Visual Diff "Codecept_IO_Screenshot_Image.png", {tolerance: 2, prepareBaseImage: false}
MisMatch Percentage Calculated is 2.85
  ✖ FAILED in 418ms


-- FAILURES:

  1) To test screen comparison with resemble Js Example test
       Compare CodeceptIO Home Page @visual-test:

      MissMatch Percentage 2.85
      + expected - actual

      -false
      +true

Codeceptjs-resemblehelper basically comes with two major functions

  1. seeVisualDiff which can be used to compare two images and calculate the misMatch percentage.
  2. seeVisualDiffForElement which can be used to compare elements on the two images and calculate misMatch percentage.

# Using Visual Knight

Visual Knight is a SaaS product which strongly supports CodeceptJS with multiple use cases. It provides an user interface to handle mismatches, statistics and more. It was designed to support Designer, Product Owner and other roles which are not familiar with coding and all this tools. All captured images are saved in a secure cloud system to not mess up your git repository.

# Setup

Create an account at Visual Knight and install the npm package

npm install @visual-knight/codeceptjs -D

# Configuring

{
    "helpers": {
        "VisualKnight": {
            "require": "@visual-knight/codeceptjs",
            "key": "YOUR_API_KEY",
            "project": "YOUR_PROJECT_ID OR PROJECT_NAME"
        }
    }
}

# Usage

/**
 * @param testName {string} Is provided to visual knight (must be unique)
 * @param options {ScreenshotOptions} Contains additional settings
 */
I.compareFullpageScreenshot(testName, options)
/**
 * @param testName {string} Is provided to visual knight (must be unique)
 * @param options {ScreenshotOptions} Contains additional settings
 */
I.compareViewportScreenshot(testName, options)
/**
 * @param cssSelector {string} Is provided to visual knight
 * @param testName {string} Is provided to visual knight (must be unique)
 * @param options {ScreenshotOptions} Contains additional settings
 */
I.compareElementScreenshot(cssSelector, testName, options)

/*
ScreenshotOptions {
  hide?: string[] // Array of css selectors which gets hidden by "opacity: 0",
  remove?: string[] // Array of css selectors which gets hidden by "display: none",
  additional?: object // Data is saved as relation to the variation. (Future: can be used for filtering)
}
*/

You can find the latest documentation here CodeceptJS helper page

# Example

Lets consider visual testing for CodeceptJS Home

Feature('To test screen comparison with Visual Knight Example test');

Scenario('Compare CodeceptIO Home Page @visual-test', async (I, adminPage) => {
    I.amOnPage("/");
    I.compareFullpageScreenshot("CodeceptIO Home Page")
});

Depending of your configuration this test will fail if no baseline exists and log the link to the image to accept or automatically accept the first run as baseline.

You can accept the first image as baseline automatically via autoBaseline: true default is false

# Using Applitools

Applitools helps Test Automation engineers, DevOps, and FrontEnd Developers continuously test and validate visually perfect mobile, web, and native apps. Now it can be used with CodeceptJS.

# Setup

Create an account at Applitools and install the npm packages

npm i codeceptjs-applitoolshelper --save
npm i [email protected] --save

# Configuring

...
  helpers: {
    WebDriver: {
      url: 'https://applitools.com/helloworld',
      browser: 'chrome',
      desiredCapabilities: {
        chromeOptions: {
          args: [ '--headless', '--disable-extensions', '--disable-gpu', '--no-sandbox', '--disable-dev-shm-usage']
        }
      },
      windowSize: '1920x600',
      smartWait: 5000,
      timeouts: {
        'script': 60000,
        'page load': 10000
      },
    },
    ApplitoolsHelper: {
      require: 'codeceptjs-applitoolshelper',
      applitoolsKey: 'YOUR_API_KEY'
    }
  },
...

# To use this helper you need to provide the following info:

  • applitoolsKey (Required): You can find your API key under the user menu located at the right hand side of the test manager toolbar
  • windowSize (Optional): the windows size as for instance 1440x700, if not provided, the default 1920x600 will be used. The windowSize will follow this precedence: ApplitoolsHelper, Webdriver.
  • appName (Optional): you can either provide your desired application name, if not provided, the default 'Application Under Test' will be used.

# Usage

/**
 * @param pageToCheck {string} Name of page you want to check
 */
I.eyeCheck(pageToCheck);

The first time you run this test a new baseline will be created, and subsequent test runs will be compared to this baseline. If any screenshot mismatch its baseline image in a perceptible way, there will be a DiffsFoundException which includes a URL that points to a detailed report where you can see the detected differences and take appropriate actions such as reporting bugs, updating the baseline and more.

-- FAILURES:

  1) Applitools functionality
       Check home page @test:
     Test 'Applitools functionality' of 'Application Under Test' detected differences!. See details at: https://eyes.applitools.com/app/batches/00000251831777088983/00000251831777088717?accountId=KO-Oh9tXI0e8VF8Ha_GLVA~~

You can find the latest documentation here Applitools Docs

# Example

Lets consider visual testing for CodeceptJS Home. You can also find example repo here: https://github.com/PeterNgTr/codeceptjs-applitoolshelper

const { I } = inject();

Feature('Applitools functionality');

Before(() => {
    I.amOnPage('https://applitools.com/helloworld');
});

Scenario('Check home page @test', async () => {
    await I.eyeCheck('Homepage');
});
Last Updated: 1/4/2020, 4:33:51 AM