Automatically create GitHub issues on failed Playwright tests

As various projects I maintain for customers include end-to-end (E2E) tests using Playwright, which run on a daily schedule and on every new release, I wanted to automate following up on failed tests. Instead of manually creating issues, I automated the process using GitHub Actions and the GitHub API. This worked great, but I thought to myself, why not share this with the world? So, I created a GitHub Action that does exactly that.

Show image

In this blog post, I will explain how you can use GitHub Action to create issues on failed Playwright tests automatically.

The GitHub Action

The GitHub Action is called Playwright Issue Creator and is available on the GitHub Marketplace.

The Playwright Issue Creator action relies on the JSON report created by the JSON reporter to verify the test results.

info

The GitHub Action only creates an issue if there is no issue open with the following name {prefix} {suite title} {test title} ({project}). If one already exists, the extension adds a comment (can be turned on/off).

Usage

Include the JSON reporter to your Playwright config

As mentioned, the GitHub Action relies on the JSON report created by the JSON reporter.

You can include the JSON reporter in your Playwright config or use the --reporter json flag when running your tests.

Using the JSON reporter in your Playwright config

1
2
3
4
5
6
7

import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["json", { outputFile: "results.json" }]
  ]
});

Using the --reporter json flag

1

PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test --reporter=json

Create a GitHub Action workflow

Create a new GitHub Action workflow in your repository and add the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

name: Playwright Tests

on:
  schedule:
    - cron: "0 6 * * *"

jobs:
  test:
    timeout-minutes: 30
    runs-on: ubuntu-latest

    # Assign write permissions for issues to the GITHUB_TOKEN
    permissions:
      contents: read
      issues: write
    
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm

      - name: Install dependencies
        run: npm ci
      
      - name: Run Playwright tests
        run: npx playwright test

      - name: Playwright issue creator
        if: always()
        uses: estruyf/[email protected]
        with:
          report-path: results.json
          issue-prefix: "E2E: "
          issue-labels: "playwright, bug"
          add-project-label: true
          add-comment: true
          job-summary: true

important

The GitHub Action requires permissions to create issues, so you need to assign those permissions to your GITHUB_TOKEN.

tip

All the configuration options are explained on the Playwright Issue Creator page.

Run your tests

After you set up the above GitHub Action workflow, the test will run daily at 6 o’clock. You can, of course, add more triggers or run the tests on every push.

Happy testing!