StorageClients w/ Multiple Crawlers
Hi!
This is my first time using Crawlee, and ... so far, so good. It's working.
However, I noticed it was using the default FileSystemStorage and creating files locally on my development machine. That's less than ideal in production.
Changing to MemoryStorageClient revealed some other problems.
I'm running multiple PlaywrightCrawlers asynchronously. The reason for that is that I want to process the scraped documents in a batch, i.e. per site.
Also, it's easier to keep things isolated that way. (Each target has it's own set of starting urls, link patterns to enqueue, and selectors to select.)
However, this fails with MemoryStorageClient because the first crawler gets the memory, and subsequent ones generate an error:
Upon investigation I discovered the docs saying:
So, even though it appears to be working with some basic tests, I'm not confident this approach will work. I actually don't want concurrent access, I want the storage to be separated, on a per-crawler basis. (Or otherwise, segmented within the Memory or File storage.) I'm not opposed to pointing it at '/tmp' in production, but the warning makes me doubtful that it would work correctly.
I did try creating multiple memory clients by setting unique queue_id, store_id and dataset_id, but that resulted in the same error.
Is this a limitation, or perhaps is there some way of doing what I'm trying to do in some other way?
Thanks for your help!
This is my first time using Crawlee, and ... so far, so good. It's working.
However, I noticed it was using the default FileSystemStorage and creating files locally on my development machine. That's less than ideal in production.
Changing to MemoryStorageClient revealed some other problems.
I'm running multiple PlaywrightCrawlers asynchronously. The reason for that is that I want to process the scraped documents in a batch, i.e. per site.
Also, it's easier to keep things isolated that way. (Each target has it's own set of starting urls, link patterns to enqueue, and selectors to select.)
However, this fails with MemoryStorageClient because the first crawler gets the memory, and subsequent ones generate an error:
Upon investigation I discovered the docs saying:
The FileSystemStorageClient is not safe for concurrent access from multiple crawler processes. Use it only when running a single crawler process at a time.
So, even though it appears to be working with some basic tests, I'm not confident this approach will work. I actually don't want concurrent access, I want the storage to be separated, on a per-crawler basis. (Or otherwise, segmented within the Memory or File storage.) I'm not opposed to pointing it at '/tmp' in production, but the warning makes me doubtful that it would work correctly.
I did try creating multiple memory clients by setting unique queue_id, store_id and dataset_id, but that resulted in the same error.
Is this a limitation, or perhaps is there some way of doing what I'm trying to do in some other way?
Thanks for your help!
This is the official developer community of Apify and Crawlee.
13,739Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
