Invalidate request queue after some time
Hello!
I would like to know if there's a builtin feature to invalidate (purge) the request queue after some time?
Thanks!
3 Replies
I don't know of such a method - you can use
RequestQueue.drop to delete a queue, but if you are in the middle of a crawl Crawlee will error out because it can't find the request queue it was using.
Why would you like to purge the queue?can you provide log or more context to properly help you??
Of course,
The goal would be to no longer process requests that have the same uniqueKey (RequestQueue behavior, if I've understood correctly?) but only for a certain period of time.
For example, if a request with uniqueKey "A" has been made, for the next 6 hours, if another request with uniqueKey "A" is added, it will be ignored
On the other hand, if it's been more than 6 hours, it will be processed.
A kind of rate limiting