continuing-cyan•16mo ago
Error accessing the requests sometimes when two crawlers are running in parallel
INFO Error: ENOENT: no such file or directory, open '~/storage/request_queues/default/iQ0kKmJLF8Tlwio.json'
Error: ENOENT: no such file or directory, mkdir ~'/storage/request_queues/default/GBHQqH6WEETimdi.json.lock'
It works most of the time, but sometimes this throws error and one of the crawler wont start.
Does anyone know what might be the reason?
3 Replies
continuing-cyanOP•16mo ago
Building on the above question, is there a way to configure different storage directory for two paralelly running crawler
it might be that they are using the same default request queue, try to pass a different request queue to the second crawler: https://crawlee.dev/api/3.8/core/class/RequestQueue
RequestQueue | API | Crawlee
Represents a queue of URLs to crawl, which is used for deep crawling of websites
where you start with several URLs and then recursively
follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
Each URL is represented using an instance of the {@apilink Request} class.
The queue can only contain...
Just use separate directories for each project.
You'll face a bunch of errors if you run them from the same directory .