Apify Discord Mirror

Updated 5 months ago

How to set depth-breadth first crawl?

At a glance

The community member suspects that the crawlee library performs a breadth-first crawl by default with enqueueLinks(), and asks if there is an option to perform a depth-first crawl instead. The comments indicate that there is no built-in way to do this, but a community member has created a feature request on the Crawlee GitHub repository. Another comment suggests that the "forefront" feature has now been added to enqueueLinks, which may address the original request.

Useful resources
I suspect that crawlee will perform a breadth first crawl by default with enqueueLinks(...)
Does exist somme option to perform a depth first?
t
L
3 comments
There unfortunately isn't a way to do this, but the idea is great! Please make a feature request on Crawlee's Github here: https://github.com/apify/crawlee/issues
Looks like the forefront feature has now been added to enqueueLinks
Add a reply
Sign up and join the conversation on Discord