Request queue taking to much space on disk?
Hi,
My request queue is taking almost as much space on disk as my scraped data (more than 100k scraped items now). The request queue takes almost 700MB. I wonder if there's a way to make it take less space on disk? I noticed there's a RequestQueueV2 but I'm not sure what benefits this brings compared to "V1".
Somehow related: is there a way to split files in multiple folders to avoid having potentially millions of files in a single folder? (this for request queues and datasets)
Any feedback would be appreciated.
Thank you!
My request queue is taking almost as much space on disk as my scraped data (more than 100k scraped items now). The request queue takes almost 700MB. I wonder if there's a way to make it take less space on disk? I noticed there's a RequestQueueV2 but I'm not sure what benefits this brings compared to "V1".
Somehow related: is there a way to split files in multiple folders to avoid having potentially millions of files in a single folder? (this for request queues and datasets)
Any feedback would be appreciated.
Thank you!
