sad-indigoS
Apify & Crawlee•3y ago•
3 replies
sad-indigo

Persist the RequestQueue (avoiding starting over)

Is it possible to persist the RequestQueue so whenever restarting a script instead of starting everything from scratch it would just keep on scrapping from the urls in the queue? I know 
CRAWLEE_PURGE_ON_START
 exists but not sure if if affects the RequestQueue. Also since an initialUrl is passed to 
await crawler.run([initialUrl])
 the idea would be to skip that in case the RequestQueue already has urls. Is that possible?
Was this page helpful?