rival-black•2y ago
How to enqueue links until a certain "depth"
I want to crawl until a certain depth of the website. Is there an option on enqueueLinks or somewhere else for this?
3 Replies
updated version
you can save the current
depth in userData of each request, increase it in the request handler when enqueuing new links and then stop when reaching the desired depthRequest | API | Crawlee
Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata.
The
Request object also stores information about errors that occurred during processing of the request.
Each Request instance has the uniqueKey property, which can be either specified
manually in the constructor or generated automaticall...limit is just limiting number of request enqueued
rival-blackOP•2y ago
Thx guys!