Vverbal-lime
Named Request Queues not getting purged
Hey folks,
I am explicitly creating request queues for my crawlers to make sure that crawler specific run options such as
1. I have tried setting the option
None of these options are working, is there some other way to purge these queues? I know its set by default to purge them but its not working for my named queues.
Also, is using queues the best way to isolate crawler specific options for each crawler? because when I used the default queue and restricted crawls to some numeric value in one crawler, and when it shut down after reaching that value, all the other crawlers would also shut down logging that max requests per crawl has been reached despite me not having specified this option when I initialized the crawlers.
I am explicitly creating request queues for my crawlers to make sure that crawler specific run options such as
maxRequestsPerCrawl1. I have tried setting the option
purgeRequestQueuecrawler.run()Did not expect property to exist, got in object `
2. setting it as a global variable in (it looks like crawlee is not picking up my crawlee.json file at all, because I tried to set logging levels in it and crawlee didnt pick it up).
3. tried using None of these options are working, is there some other way to purge these queues? I know its set by default to purge them but its not working for my named queues.
Also, is using queues the best way to isolate crawler specific options for each crawler? because when I used the default queue and restricted crawls to some numeric value in one crawler, and when it shut down after reaching that value, all the other crawlers would also shut down logging that max requests per crawl has been reached despite me not having specified this option when I initialized the crawlers.
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
