Vverbal-lime
Named Request Queues not getting purged
Hey folks,
I am explicitly creating request queues for my crawlers to make sure that crawler specific run options such as
1. I have tried setting the option
None of these options are working, is there some other way to purge these queues? I know its set by default to purge them but its not working for my named queues.
Also, is using queues the best way to isolate crawler specific options for each crawler? because when I used the default queue and restricted crawls to some numeric value in one crawler, and when it shut down after reaching that value, all the other crawlers would also shut down logging that max requests per crawl has been reached despite me not having specified this option when I initialized the crawlers.
I am explicitly creating request queues for my crawlers to make sure that crawler specific run options such as
maxRequestsPerCrawlmaxRequestsPerCrawl can be set on a crawler to crawler basis, but the issue with this approach is that the request queues are not getting purged after every crawl, resulting in the crawlers resuming the session from before. These are the approaches I tried1. I have tried setting the option
purgeRequestQueuepurgeRequestQueue to true explicitly in the crawler.run()crawler.run() func but it results in this errorDid not expect property Did not expect property purgeRequestQueue to exist, got to exist, got true in object in object options`
2. setting it as a global variable in `
2. setting it as a global variable in crawlee.json (it looks like crawlee is not picking up my crawlee.json file at all, because I tried to set logging levels in it and crawlee didnt pick it up).
3. tried using (it looks like crawlee is not picking up my crawlee.json file at all, because I tried to set logging levels in it and crawlee didnt pick it up).
3. tried using await purgeDefaultStorages()` in my entry fileNone of these options are working, is there some other way to purge these queues? I know its set by default to purge them but its not working for my named queues.
Also, is using queues the best way to isolate crawler specific options for each crawler? because when I used the default queue and restricted crawls to some numeric value in one crawler, and when it shut down after reaching that value, all the other crawlers would also shut down logging that max requests per crawl has been reached despite me not having specified this option when I initialized the crawlers.
