NeoNomadeN
Apify & Crawlee3y ago
10 replies
NeoNomade

Blocking certain requests

I'm trying to block some requests in Puppeteer but it doesn't seem to work if I run the script headed :
const blockedResourceTypes = ['webp', 'svg', 'mp4', 'jpeg', 'gif', 'avif', 'font']
const crawler = new PuppeteerCrawler({
    launchContext: {
        launchOptions: {
            headless: false,
            devtools: true,
            defaultViewport:{ width: 1920, height: 6000 },
            args: [
                '--disable-dev-shm-usage',
            ]
        },
        useIncognitoPages: true,
    },
    proxyConfiguration,
    requestHandler: router,
    maxConcurrency: 16,
    maxRequestRetries: 15,
    maxRequestsPerMinute: 2,
    navigationTimeoutSecs: 120,
    useSessionPool: true,
    failedRequestHandler({ request }) {
        log.debug(`Request ${request.url} failed 15 times.`);
    },

    preNavigationHooks: [
        async ({ addInterceptRequestHandler }) => {
            await addInterceptRequestHandler((request) => {
                if (blockedResourceTypes.includes(request.resourceType())) {
                    return request.respond({
                        status: 200,
                        body: 'useless shit',
                    });
                }
                return request.continue();
            });
        },
    ],
});


Any ideas ?
Was this page helpful?