plain-purple•2y ago
Purge on start
If I use CRAWLEE_PURGE_ON_START=0, will the crawler revisit again all the urls or the only the ones that are left over?
I mean those which were not flagged as error but added to the request queue.
Does the request queue reset?
1 Reply
var Defines whether to purge the default storage folders before starting the crawler run.
So if 0 it will continue crawling with URLs that were left in your queue.