Eefficient-indigo
Limiting Number of Concurrent Agents?
Hi -- I'm just starting to work with the platform. I've been very impressed so far but I'm running into a challenge constraining things so I don't get a lot of what are effectively "out of resources" errors.
What I am doing is trying to use Apify as the crawler end of an airtable knowlege tracking tool we use internally. Users submit links into an Airtable, we pull the links into a cannon including content with Zapier as the orchestration tool. Apify is much better at the pulling in part, but the challenge I'm running into is we get a lot of errors because this is a bursty process and we request too many page parsings at the same time. Zapier doesn't seem to have a way to do this elegantly but I'm hoping Apify does.
Ideally there is a setting somewhere that says "only run up to 4 concurrent agents at once for this task" or "queue agents by available RAM on the account." Is there any way to approach this, especially from a no-code solution angle?
What I am doing is trying to use Apify as the crawler end of an airtable knowlege tracking tool we use internally. Users submit links into an Airtable, we pull the links into a cannon including content with Zapier as the orchestration tool. Apify is much better at the pulling in part, but the challenge I'm running into is we get a lot of errors because this is a bursty process and we request too many page parsings at the same time. Zapier doesn't seem to have a way to do this elegantly but I'm hoping Apify does.
Ideally there is a setting somewhere that says "only run up to 4 concurrent agents at once for this task" or "queue agents by available RAM on the account." Is there any way to approach this, especially from a no-code solution angle?
This is the official developer community of Apify and Crawlee.
13,923Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
