`maxRequestsPerMinute` But for session

:perfecto: Hey!

Firstly I just want to thank you for creating such an amazing product

!

Question itself:

Regarding documentation (https://crawlee.dev/docs/guides/scaling-crawlers) we can set

maxRequestsPerMinute

maxRequestsPerMinute

limitation for global crawler process.
But for some cases,

maxRequestsPerMinute

maxRequestsPerMinute

should be set by session itself.

For example:
Website

iammastrongwebsite.com

iammastrongwebsite.com

have request limit per session (5rpm).

I have a bunch of proxies, and setting

maxRequestsPerMinute

maxRequestsPerMinute

not ideal, since all of my imaginative hundreds of proxies will be waiting without providing any payload

Of course, we can remember simple math, and make
maxRequestsPerMinute = [SessionLimit] * 5
maxRequestsPerMinute = [SessionLimit] * 5
, but it's actually could be worse, since this approach full breaks a purpose of autoscaling

Suggestion:
Add ability to set

maxRequestsPerMinute

maxRequestsPerMinute

in sessions itself.

Thanks again

Scaling our crawlers | Crawlee · Build reliable crawlers. Fast.

Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

Hall•11/23/24, 3:57 AM

Someone will reply to you shortly. In the meantime, this might help:

Exp•11/23/24, 12:26 PM

Hi, you can try to use rotate proxy

rubber-blueOP•12/19/24, 12:43 PM

@Oleg V. Hey! Thanks for your reaction! Question not resolved yet. Can you please unmark it?

Uunexpected-ivory Hi, you can try to use rotate proxy

rubber-blueOP•12/19/24, 12:45 PM

Hey! Thanks for your answer. It is of one possible workarounds... But again, it will work badly, if you have session-like approach to scrape resources