How to avoid requesting some static resources?

When crawling with

Playwright

Playwright

Playwright

Playwright or

Puppeteer

Puppeteer

Puppeteer

Puppeteer, a lot of static assets (eg js, css, png, jpg) are loaded.

Is it possible to only request static resources for the first time, and use the last cached data for the next crawling without making a request.

MEE6•11/8/22, 3:09 PM

@Dillian just advanced to level 1! Thanks for your contributions!

Oleg V.•11/8/22, 3:21 PM

You can create an array of resourceTypes that you'd like to block.

Example for Playwright:
const BLOCKED = ['image', 'stylesheet', 'media', 'font','other'];const BLOCKED = ['image', 'stylesheet', 'media', 'font','other'];

Then within your preNavigationHooks preNavigationHooks of your crawler, add this function:

async ({ page }) => {
    await page.route('**/*', (route) => {
        if (BLOCKED.includes(route.request().resourceType())) return route.abort();
        return route.continue()
    });
};

async ({ page }) => {
    await page.route('**/*', (route) => {
        if (BLOCKED.includes(route.request().resourceType())) return route.abort();
        return route.continue()
    });
};

Or you can try to use Crawlee util functions (also in preNavigationHooks preNavigationHooks option):
https://crawlee.dev/api/3.0/playwright-crawler/namespace/playwrightUtils#blockRequests
https://crawlee.dev/api/3.0/puppeteer-crawler/namespace/puppeteerUtils#blockRequests

ill-bronzeOP•11/9/22, 3:54 PM

@Oleg V. thanks for help

。Is it possible to only request static resources for the first time, and use the last cached data for the next response without making a request?

Lukas Krivka•11/14/22, 11:58 AM

I think that is what most browsers do by default (including if run in Crawlee). If not for those specific cases, you can maintain your own cache object and intercept requests to feed them cached response