Injecting Axe a11y tester

I would like to use Crawlee to crawl a bunch of internal sites and run the Axe accessibility scanner on each page. I figured out how to inject the script they reference in their getting started docs (https://github.com/dequelabs/axe-core#getting-started) using the

page.addInitScript

page.addInitScript

import { PlaywrightCrawler, Dataset } from 'crawlee';
import axe from 'axe-core';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        await page.addInitScript('./node_modules/axe-core/axe-min.js');

        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        const results = await axe.run();
        console.log(results.violations)

        await Dataset.pushData({ title, url: request.loadedUrl });

        await enqueueLinks();
    },
});

await crawler.run(['https://dequeuniversity.com/demo/mars/']);

import { PlaywrightCrawler, Dataset } from 'crawlee';
import axe from 'axe-core';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        await page.addInitScript('./node_modules/axe-core/axe-min.js');

        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        const results = await axe.run();
        console.log(results.violations)

        await Dataset.pushData({ title, url: request.loadedUrl });

        await enqueueLinks();
    },
});

await crawler.run(['https://dequeuniversity.com/demo/mars/']);

But every page after that throws this error.

INFO PlaywrightCrawler: Starting the crawl
INFO PlaywrightCrawler: Title of https://dequeuniversity.com/demo/mars/ is 'Mars Commuter: Travel to Mars for Work or Pleasure!'
WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Required "window" or "document" globals not defined and cannot be deduced from the context. Either set the globals before running or pass in a valid Element. {"id":"9syPc5JbUuAjPx1","url":"https://dequeuniversity.com/demo/mars/","retryCount":1}

If I comment out the call to axe.run() the error goes away and things 'work'.

Any idea what could be causing this?

Mars Commuter: Travel to Mars for Work or Pleasure!

MarsCommuter - Your gateway to the Red Planet

GitHub

GitHub - dequelabs/axe-core: Accessibility engine for automated Web...

Accessibility engine for automated Web UI testing. Contribute to dequelabs/axe-core development by creating an account on GitHub.

sour-pinkOP•9/15/22, 8:34 PM

My code was wrong. This should be a better example

sour-pinkOP•9/15/22, 8:36 PM

Ahh, and I needed to add a try/catch to log the error that was being hidden from stdout. Nevermind, nothing to see here :S

slow-yellow•9/15/22, 8:38 PM

I would not recommend putting the

page.addInitScript

page.addInitScript

in the

requestHandler

requestHandler

because

requestHandler

requestHandler

runs once the

load

load

event has fired. Try adding that into

preNavigationHooks

preNavigationHooks

instead like this:

The error you're receiving is because the axe script was never run on the page since it had already been initialized.

slow-yellow•9/15/22, 8:38 PM

If the above doesn't work, you can try this hacky solution I used a while back to run an init script. This registers the init script at the browser level to ensure it is ALWAYS run before any other scripts on the page:

slow-yellow•9/15/22, 8:38 PM

Second one should work 100%

slow-yellow•9/15/22, 8:39 PM

Hope this helps!

sour-pinkOP•9/16/22, 2:43 PM

I tried the second one and after wrapping the call to in a try/catch I get this error.

Error: Required "window" or "document" globals not defined and cannot be deduced from the context. Either set the globals before running or pass in a valid Element.
at setupGlobals (/Users/LGoolsby/dev/playground/axe/sample/node_modules/axe-core/axe.js:21231:15)
at Object.run4 [as run] (/Users/LGoolsby/dev/playground/axe/sample/node_modules/axe-core/axe.js:21486:7)
at PlaywrightCrawler.requestHandler [as userProvidedRequestHandler] (file:///Users/LGoolsby/dev/playground/axe/sample/src/main.ts:43:39)
at async wrap (/Users/LGoolsby/dev/playground/axe/sample/node_modules/@apify/src/index.ts:77:27)

MEE6•9/16/22, 2:43 PM

@Lane Goolsby just advanced to level 1! Thanks for your contributions!

sour-pinkOP•9/16/22, 2:53 PM

Same error with the first one as well

sour-pinkOP•9/16/22, 3:03 PM

The line in the axe.js that's blowing chunks is doing this:

sour-pinkOP•9/16/22, 3:16 PM

This seems relevant https://github.com/apify/crawlee/pull/1337

GitHub

feat: add `playwright.utils.injectJQuery` by barjin · Pull Request ...

closes #1336
JQuery requires the global document object, which is not available when Page.addInitScript-s are added. This solution, therefore, introduces the new injectFile option waitForDOM, which...

sour-pinkOP•9/16/22, 6:24 PM

I figured it out! Here's a working solution.

slow-yellow•9/16/22, 6:47 PM

Super nice!! Running the script after the page has loaded seems to also work