equal-aqua•4y ago

How to set depth-breadth first crawl?

I suspect that crawlee will perform a breadth first crawl by default with enqueueLinks(...) Does exist somme option to perform a depth first?

3 Replies

passive-yellow•4y ago

There unfortunately isn't a way to do this, but the idea is great! Please make a feature request on Crawlee's Github here: https://github.com/apify/crawlee/issues

GitHub

Issues · apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast. - Issues · apify/crawlee

equal-aquaOP•4y ago

Made: https://github.com/apify/crawlee/issues/1595

GitHub

Option to perform a depth first crawl · Issue #1595 · apify/crawlee

Which package is the feature request for @crawlee/core Feature Crawlee will perform a breadth first crawl by default with enqueueLinks(...). Make some option to perform a depth first. Ref Wikipedia...

passive-yellow•4y ago

Looks like the forefront feature has now been added to enqueueLinks

How to set depth-breadth first crawl?

Did you find this page helpful?