Skip to content

analyze

Uses AI to analyze test failures and provide insights

This plugin analyzes failed tests using AI to provide detailed explanations and group similar failures. When enabled with —ai flag, it generates reports after test execution.

Usage

Section titled “Usage”
// in codecept.conf.js
exports.config = {
  plugins: {
    analyze: {
      enabled: true,
      clusterize: 5,
      analyze: 2,
      vision: false
    }
  }
}

Configuration

Section titled “Configuration”
  • clusterize (number) - minimum number of failures to trigger clustering analysis. Default: 5
  • analyze (number) - maximum number of individual test failures to analyze in detail. Default: 2
  • vision (boolean) - enables visual analysis of test screenshots. Default: false
  • categories (array) - list of failure categories for classification. Defaults to:
    • Browser connection error / browser crash
    • Network errors (server error, timeout, etc)
    • HTML / page elements (not found, not visible, etc)
    • Navigation errors (404, etc)
    • Code errors (syntax error, JS errors, etc)
    • Library & framework errors
    • Data errors (password incorrect, invalid format, etc)
    • Assertion failures
    • Other errors
  • prompts (object) - customize AI prompts for analysis
    • clusterize - prompt for clustering analysis
    • analyze - prompt for individual test analysis

Features

Section titled “Features”
  • Groups similar failures when number of failures >= clusterize value
  • Provides detailed analysis of individual failures
  • Analyzes screenshots if vision=true and screenshots are available
  • Classifies failures into predefined categories
  • Suggests possible causes and solutions

Parameters

Section titled “Parameters”
  • config Object Plugin configuration

Returns void