Crawlee concurrency with multiple domains in one crawler?
If I run a single PlaywrightCrawler that crawls multiple different websites, are concurrency and rate limits shared across all domains, or does Crawlee track them separately per domain?
I’m asking because I’m building a general crawler that scrapes schema.org markup from lots of unrelated sites, and the crawl logic is almost identical, so I’m using one crawler instance instead of one per domain. I want to understand whether domains compete for the same concurrency slots or not? Basically, what I'm asking is if I set
I’m asking because I’m building a general crawler that scrapes schema.org markup from lots of unrelated sites, and the crawl logic is almost identical, so I’m using one crawler instance instead of one per domain. I want to understand whether domains compete for the same concurrency slots or not? Basically, what I'm asking is if I set
minConcurrencyminConcurrency to 3, is it 3 concurrent requests for website_a.com and 3 concurrent requests website_b.com, or is it 3 concurrent requests spread out between website_a.com and website_b.com?Solution
or is it 3 concurrent requests spread out between website_a.com and website_b.com?
This option is correct for crawlee.
