What optimizations work for you?
I'm attempting to use crawlee and puppeteer to crawl between 15 and 30 million urls. I'm not rich but I also can't wait forever for the crawl to finish, so I've spent some time over the last few days hunting for different optimizations that might make my crawler faster. This is more challenging that usual when you're crawling a laundry list of unknown sites.
First, here's some of the code I'm working with at this point. To get this running you just:
I'm then working off the default typescript puppeteer template by selecting it after running this command:
And in the next response I'll post some of the code for setting up my crawler with as many optimizations as I've found useful. Running out of allowed characters...
First, here's some of the code I'm working with at this point. To get this running you just:
npm install crawlee puppeteer-extra puppeteer-extra-stealth @sparticuz/chromium puppeteer-core
I'm then working off the default typescript puppeteer template by selecting it after running this command:
npx crawlee create your-project-name
And in the next response I'll post some of the code for setting up my crawler with as many optimizations as I've found useful. Running out of allowed characters...
