Bbrilliant-lime
How to instantiate 1 crawler, and run it with incoming incoming requests
So, I'm trying to understand what I'm doing architecturally wrong. I define a "crawler" using Playwright crawler. Then, I add some urls to the requestQueue and it runs fine. I can then load more into the crawler, and again call run and it works. But, in the case where my crawler will have a queue that is sending it new requests ongoing, i'm not sure how to architect this.
When you make it where each crawler receive a url, processes it, and closes down, it makes it impossible effectively to run things in parallel.
When you try to add things to the queue while the crawler is running (using addRequests), that seems to fail as well.
So, how do I architect this?
This is my example code for reference.
(I attempted to add the example code i'm using, but it was too long. So, here's a gist: https://gist.github.com/wflanagan/2ea9316db8d3173f5ad3fbde2443ca67)
When you make it where each crawler receive a url, processes it, and closes down, it makes it impossible effectively to run things in parallel.
When you try to add things to the queue while the crawler is running (using addRequests), that seems to fail as well.
So, how do I architect this?
This is my example code for reference.
(I attempted to add the example code i'm using, but it was too long. So, here's a gist: https://gist.github.com/wflanagan/2ea9316db8d3173f5ad3fbde2443ca67)
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
