Apify Discord Mirror

Updated 6 hours ago

Max Depth option

Hello! Just wondering whether it is possible to set max depth for the crawl?
Previous posts (2023) seems to make use of 'userData' to track the depth.
Thank you.
В
R
2 comments
Hi, this can be set. You can add a maxdepth parameter in the INPUT_SCHEMA.json file, then retrieve it from the code and implement the specified depth functionality.
@mjh Hey, when using python, you can use max_crawl_depth (https://crawlee.dev/python/api/class/BasicCrawlerOptions#max_crawl_depth) .
Otherwise, you can track the current depth using request.userData. When adding new links in the request handler, just increment the depth by one and use that for each newly enqueued request (https://crawlee.dev/api/core/class/Request)
Add a reply
Sign up and join the conversation on Discord