Using Lighthouse CI with GitHub Actions

Jul 15, 2021

This article explains the steps to integrate Lighthouse into your pull request process using GitHub Actions. Lighthouse is a web page auditing tool, and GitHub Actions provides GitHub users a way to perform continuous integration (“CI”) tasks right within GitHub itself.

By combining them, you can automatically generate Lighthouse audit reports showing how changes proposed in pull requests will affect your app's most important pages.

Steps

Pick what pages to audit
Run Lighthouse from Google Chrome
Run Lighthouse from the CLI
Run Lighthouse from GitHub Actions
Add the Lighthouse CI GitHub App
Merge Your Changes

Step 1: Pick What Pages to Audit 📋

Before I start auditing pages, I need to figure out what pages to audit. This should be informed by organization goals for how visitors should use the site.

I’d take a few minutes to consider:

Who visits the site
What goals those visitors probably have
What pages they’d need to visit in order to achieve that goal

If my site has different types of visitors with different goals, I’ll make a list for each of them. If working with teammates (especially ones who are more explicitly tasked with advocating for different types of visitors) this is a good time to get their input.

Once I’ve got a list of my important use cases and associated pages, I’ll store my notes somewhere convenient for reviewing later, since even outside the context of this exercise, it provides useful insight on how the org thinks about users. For my immediate purposes, I’ll make a plaintext list of all the pages identified, write them as relative URLs, and remove any duplicates.

By the end of this step, you’ll have a list of URLs for your site’s most important pages.

Step 2: Run Lighthouse from Google Chrome 🔬

At this point, I use Chrome DevTools Lighthouse tab to manually audit those pages on a local server, opening each page one by one, opening Chrome DevTools, switching to the Lighthouse tab, and running audits for performance, accessibility, best practices, and SEO using the “mobile device” setting.

I could skip this step, but I feel better when I run my audits manually and see the results with a nice in-browser UI.

Tips

Make sure you’re not testing your app server in “dev mode” with tooling like Browsersync running. Test as closely to production as is practical.
Some server applications and environments use proprietary config files that your local web server may not recognize. For example, Netlify looks for a netlify.toml file to set things like CSP headers. Your local web server may not know to use those files.
If you see a score that looks like a crazy outlier, look into it. Check for possible server misconfiguration. Run a couple of test runs and prepare to do some local server debugging.
Especially if you’re just getting started with Lighthouse, you’re probably going to uncover some quick and easy things you can do to improve your Lighthouse score. That’s great! Make notes of them, but stay on target: finish setting up your Lighthouse automation first.

By the end of this step, you’ll have some Lighthouse audit scores for your most important pages.

Step 3: Run Lighthouse CI from the CLI 🖥

After learning how to get my server prepared for audits and seeing how my pages score, I’ll get Lighthouse locally from the CLI the same way I’ll be doing with GitHub Actions.

Step 3.1: Install LHCI's CLI tool

npm install -g @lhci/[email protected]

You don’t necessarily have to use sudo to install a package globally. Sindre Sorhus explains how in this guide.

Step 3.2: Confirm it's installed globally

lhci --version

This should return something like:

0.7.2

Step 3.3: Write a configuration file

LHCI will look for a file named lighthouserc.js, which should tell LHCI to:

Upload all reports to “temporary public storage,” Google’s free and publicly accessible option. If working on a “secret” project or one involving sensitive data, there are other options that may be better.
Collect audit data using a static web server hosting the assets in the dist/ directory
Collect audit data from these URLs: http://localhost/, http://localhost/articles/, and http://localhost/code-projects/

Example `lighthouserc.js`:

module.exports = {
  ci: {
    upload: {
      target: 'temporary-public-storage',
    },
    collect: {
      staticDistDir: './dist',
      url: [
        'http://localhost/',
        'http://localhost/articles/',
        'http://localhost/code-projects/',
      ],
    },
  },
}

Test Faster! ⚡️

Even though I showed my configuration file testing three different pages, I recommend reducing your cycle time by auditing only one or two pages while setting this up. You can add the rest once everything is working.

Step 3.4: Build the app

For my example, I'm working with a static site generator that I can build by running npm run generate which puts all the built assets into a dist/ directory. This will vary from project to project! If I’m testing an app that isn’t running on a static site generator, I’ll use the startServerCommand to tell Lighthouse CI how to start my server.

Step 3.5: Run Lighthouse CI and confirm it worked

lhci autorun

Confirm it worked right:

Verify LHCI finds the configuration file, which it reports:

✅  Configuration file found

Verify it collects audits on the pages specified in the configuration file. By default, LHCI will run 3 times and report the median score:

Running Lighthouse 3 time(s) on http://localhost:56251/
Run #1...done.
Run #2...done.
Run #3...done.
Done running Lighthouse!

Verify the audit reports are published to publicly-accessible URLs

Uploading median LHR of http://localhost:56251/...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1626375889671-9480.report.html

By the end of this step, you’ll have the ability to run LHCI with your project’s specific settings from the command line.

Step 4: Run Lighthouse CI from GitHub Actions 🏗

Now it’s time to get all this automated. If I’m working on a new project that isn’t already using GitHub Actions, I’ll make a new directory: .github/actions/ and put a file ci.yml in it. If the project already uses GitHub Actions, I’ll need to decide between updating an existing workflow with a new Lighthouse job or updating an existing job with a new Lighthouse step. Whoever’s responsible for guiding the project’s testing process should be able to inform this decision.

Since this Lighthouse audit is an end-to-end test, it should theoretically be run after ensuring your code passes lighter/quicker tests. If I’ve got linting and unit tests and integrated tests already running in a single job, I prefer to put the Lighthouse step at the end.

Example `ci.yml`:

name: End-to-End Tests
on: [push]
jobs:
  lhci:
    name: Lighthouse
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Set up Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - name: Install dependencies and build
        run: |
          npm install
          npm run build
      - name: Test site with Lighthouse CI
        run: |
          npm run generate # Or whatever will build the project
          npm install -g @lhci/[email protected]
          lhci autorun

I’ll commit this file to my repo on a new branch and open a pull request. In that pull request’s “Checks” tab, after a moment I should see a log showing GitHub Actions running my Lighthouse audits. If I need to change anything in the workflow, GitHub will use workflow files on my branch itself to guide its PR’s checks. For a faster review cycle when troubleshooting, I’ll use the act library to run GitHub Actions locally.

By the end of this step, you’ll have an open pull request with some automated checks showing an output log pretty close to what you saw when you ran the tests locally.

Step 5: Add the Lighthouse CI GitHub App ✨

Finally, to view my Lighthouse scores as checks without needing to go to the “Checks” tab, I can install the Lighthouse CI GitHub App.

Install the Lighthouse CI GitHub App and grant it permission to access my app’s repository
After authorizing Lighthouse CI, I’ll see a confirmation screen with an access token string that will look like 82cgwlT9UrQSWfV:18297505:r5SnfRqsN. If I lose it, I’ll have to get a new one by uninstalling and reinstalling the Lighthouse CI GitHub App.
Add my access token to my GitHub repo’s secrets
1. In the GitHub UI, I’ll go to my repository and click the “Settings” tab
2. Choose “Secrets”
3. Choose “New Repository Secret”
4. Name the secret LHCI_GITHUB_APP_TOKEN and set its value using the token from that authorization page.
Pass a reference to that secret in my GitHub Actions workflow so the step to run Lighthouse has access to it:

Example `ci.yml`:

...
      - name: Run Lighthouse
        run: |
          npm run generate
          npm install -g @lhci/[email protected]
          lhci autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
...

I’m showing my run commands to give some context for where exactly I put the reference in my own workflow, but your run commands will likely be a little different. The important part you’ll be copying and pasting is that last env part where I set LHCI_GITHUB_APP_TOKEN using GitHub’s syntax for referencing secrets.

If you see a message in your logs that says it successfully updated the status check for GitHub but you’re not seeing a new status check on your pull request, confirm your workflow’s checkout step checks out the same hash your branch is pointed at. For troubleshooting, the official GitHub checkout action offers “Checkout out the ref” output in its log:

Checking out the ref
/usr/bin/git log -1 --format='%H'
'5af80f0a7fda021196b7adedc63bb06ce865e27e'

Compare that third line with your most recent commit hash and ensure it's not different. I don’t understand why this sometimes happens, but it’s a known issue. If the hash listed doesn’t match the commit your branch is pointed at, use this fix:

Example `ci.yml`:

...
      - name: Checkout 🛎
        uses: actions/checkout@master
        with:
          ref: ${{ github.event.pull_request.head.sha }}
...

By the end of this step, you’ll see your site’s Lighthouse scores from your GitHub Pull Request’s “Conversation” tab under where it shows “Checks.”

Step 6: Merge Your Changes 🚢

If you took my advice earlier about only auditing one or two pages while getting this set up, now's a good time to add the rest of the pages. Assuming everything still works, celebrate and merge your changes! 🍾

Navigation Event	Duration
Process Unload Event	unknown
Handle Redirects	unknown
Service Worker Init	unknown
Service Worker Fetch Event	unknown
HTTP Cache	unknown
DNS Lookup	unknown
TCP Handshake	unknown
Client Request	unknown
Server Response	unknown
Processing Page	unknown
Run "Load" Event Handlers	unknown
Total	unknown
* You may notice the total of these events is different than the total amount of time the page took to load. I haven't found an official explanation, but my best guess is that sometimes your browser spends time doing stuff between the steps listed above.

Using Lighthouse CI with GitHub Actions

Steps

Step 1: Pick What Pages to Audit 📋

Step 2: Run Lighthouse from Google Chrome 🔬

Tips

Step 3: Run Lighthouse CI from the CLI 🖥

Step 3.1: Install LHCI's CLI tool

Step 3.2: Confirm it's installed globally

Step 3.3: Write a configuration file

Example lighthouserc.js:

Test Faster! ⚡️

Step 3.4: Build the app

Step 3.5: Run Lighthouse CI and confirm it worked

Step 4: Run Lighthouse CI from GitHub Actions 🏗

Example ci.yml:

Step 5: Add the Lighthouse CI GitHub App ✨

Example ci.yml:

Example ci.yml:

Step 6: Merge Your Changes 🚢

Example `lighthouserc.js`:

Example `ci.yml`:

Example `ci.yml`:

Example `ci.yml`: