Ways to minimize traffic (save money) when crawling-scraping?

sacred-emerald · 2023-01-06T18:23:22.724Z

**1. Block images, media files and similar things** It can be done either with `preNavigationHooks`, see https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#preNavigationHooks or with the `blockRequests` https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlingContext#blockRequests As far as I know, `blockRequests` has some limitations (does it works in incognito mode with Firerox as launcher?). This was discussed in this forum, see: https://discord.com/channels/801163717915574323/1039557325784105002 https://discord.com/channels/801163717915574323/1019949012415160370 **2. Use cache** As far as I understand - you can not have both: cache AND incognito mode. Well, there is the `experimentalContainers` thing - in theory it should allow both cache and incognito. I tried it, see https://discord.com/channels/801163717915574323/1060738415370453032/1060952860868739192 it looks it's not really "incognito" when fingerprint.com recognize you even when your IP is different. (you can disprove me - may be my test was wrong, who knows?) **3. Something else to reduce traffic?** Please suggest... **4. Actually I care more about money than about traffic...** So one of the ideas - to use "Datacenter proxy" instead of "Residential"... I see Datacenter proxies for about $0.7 per GB - much cheaper that Residential. Does it make sense to try? What is your experience with Datacenter proxies ?

M

MEE6•1/6/23, 6:23 PM

@new_in_town just advanced to level 4! Thanks for your contributions!

A

awake-maroon•1/6/23, 6:50 PM

Why not change set of datacenter proxies every 5-6 days?

L

Lukas Krivka•1/9/23, 10:19 PM

blockRequests doesn't work in Firefox at all sadly
Generally, you reduce traffic and compute by scraping via API which is possible on most modern websites. Those who have all in HTML are heavier but you almost never need browser. https://developers.apify.com/academy/api-scraping
Browser is so much heavier that none of the above things really matter if you don't need it

Apify

API scraping · Apify Developers

Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.

Cclean-aquamarine 1. blockRequests doesn't work in Firefox at all sadly 2. Generally, you reduce ...

S

sacred-emeraldOP•1/9/23, 10:35 PM

by the way:

I want to avoid downloading unnecessary files (unnecessary requests) so I'm using
the method described here:
crawlee-jsHow to avoid requesting some static resources?

Here https://playwright.dev/docs/api/class-request#request-resource-type (Playwright documentation) is the list of resource types to check:

document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

BUT!
Looking here
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/ResourceType
I see much much more resource types. Things like

"csp_report", "beacon", "imageset", "ping"

"csp_report", "beacon", "imageset", "ping"

"csp_report", "beacon", "imageset", "ping"

"csp_report", "beacon", "imageset", "ping"

and many many others.

Should I include elements both lists in my BLOCKED array? (Imagine I'm paranoid and want to block everything except the main document)

webRequest.ResourceType - Mozilla | MDN

This type is a string, which represents the context in which a resource was fetched in a web request.

Request | Playwright

Whenever the page sends a request for a network resource the following sequence of events are emitted by [Page]:

Nnew_in_town by the way: I want to avoid downloading unnecessary files (unnecessary requests...

L

Lukas Krivka•1/9/23, 11:05 PM

Sure, why not. Just keep in mind that the website might not work properly if you block some stuff.

S

sacred-emeraldOP•1/25/23, 9:46 AM

...and here the resource types I block:

const BLOCKED_IMG        =                                                               ['image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS    =                                                 ['stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS_JS = ['websocket', 'xhr', 'xmlhttprequest', 'script', 'stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG        =                                                               ['image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS    =                                                 ['stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS_JS = ['websocket', 'xhr', 'xmlhttprequest', 'script', 'stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG        =                                                               ['image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS    =                                                 ['stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS_JS = ['websocket', 'xhr', 'xmlhttprequest', 'script', 'stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG        =                                                               ['image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS    =                                                 ['stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

const BLOCKED_IMG_CSS_JS = ['websocket', 'xhr', 'xmlhttprequest', 'script', 'stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];

For a new site I try to block everything in BLOCKED_IMG_CSS_JS.
In case the site does not work (pages not rendered properly) I try the BLOCKED_IMG_CSS. If it is still not work - BLOCKED_IMG and than - no block at all.

Feel free to use/improve this approach.
Share your improvements :-)

S

sacred-emeraldOP•1/25/23, 9:53 AM

and the code that implements blocking:

    preNavigationHooks: [

        async ({ page, request }) => {
            await page.route('**/*', (route) => {
                if ( (request.userData.headLessImg==='noimg') &&  (BLOCKED_IMG.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocss') &&  (BLOCKED_IMG_CSS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocssnojs') &&  (BLOCKED_IMG_CSS_JS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                return route.continue();
            });
        },
    ],

    preNavigationHooks: [

        async ({ page, request }) => {
            await page.route('**/*', (route) => {
                if ( (request.userData.headLessImg==='noimg') &&  (BLOCKED_IMG.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocss') &&  (BLOCKED_IMG_CSS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocssnojs') &&  (BLOCKED_IMG_CSS_JS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                return route.continue();
            });
        },
    ],

    preNavigationHooks: [

        async ({ page, request }) => {
            await page.route('**/*', (route) => {
                if ( (request.userData.headLessImg==='noimg') &&  (BLOCKED_IMG.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocss') &&  (BLOCKED_IMG_CSS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocssnojs') &&  (BLOCKED_IMG_CSS_JS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                return route.continue();
            });
        },
    ],

    preNavigationHooks: [

        async ({ page, request }) => {
            await page.route('**/*', (route) => {
                if ( (request.userData.headLessImg==='noimg') &&  (BLOCKED_IMG.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocss') &&  (BLOCKED_IMG_CSS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                if ( (request.userData.headLessImg==='noimgnocssnojs') &&  (BLOCKED_IMG_CSS_JS.includes(route.request().resourceType())) )
                {
                    return route.abort();
                }
                return route.continue();
            });
        },
    ],

So I set the

request.userData.headLessImg

request.userData.headLessImg

request.userData.headLessImg

request.userData.headLessImg per request and the code in

preNavigationHooks

preNavigationHooks

just check the value....

Ssacred-emerald and the code that implements blocking: ``` preNavigationHooks: [ a...

A

awake-maroon•2/1/23, 7:07 PM

Great thanks for sharing. If possible Can you please respond on DM?

Aawake-maroon Great thanks for sharing. If possible Can you please respond on DM?

S

sacred-emeraldOP•2/4/23, 8:32 AM

write DM again (can not see it now)

M

MEE6•2/4/23, 8:32 AM

@new_in_town just advanced to level 6! Thanks for your contributions!