CWhy request_queues's metadata file is not cleaned when purge_on_start=True?
Hello,
I find this behavior odd that when
What is the reason behind this? Which use-case this covers?
If you run the repro code (it crawls 2 URLs) multiple times right after another and check the log/metadata json. I see the following:
The 2nd run 0/4 is misleading because the only two requests were scheduled and the previous one has been purged.
After the 2nd run the manifest.json content contains handled_request_count: 4 and total_request_count: 4 which is printed to the logs.
My expectation would be 2 for both values
I've attached repro code as a file due to post length limitation
Thank you,
CL
I find this behavior odd that when
Configuration(purge_on_start=True)storage/request_queues/default/__metadata__.jsoncrawlee.storage_clients._file_system._request_queue_client.FileSystemRequestQueueClient.purgeWhat is the reason behind this? Which use-case this covers?
If you run the repro code (it crawls 2 URLs) multiple times right after another and check the log/metadata json. I see the following:
The 2nd run 0/4 is misleading because the only two requests were scheduled and the previous one has been purged.
After the 2nd run the manifest.json content contains handled_request_count: 4 and total_request_count: 4 which is printed to the logs.
My expectation would be 2 for both values
I've attached repro code as a file due to post length limitation
Thank you,
CL
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
