Cena AshooriC
Apify & Crawlee•5mo ago•
5 replies
Cena Ashoori

Crawler become slower as time goes on

Hello guys, thanks for your great tools. I have a problem with crawlee, it works well when I run crawler at the beginning, but when my vpn has a problem and I switch my config, crawler won't continue and I have to restart it. Are there any timeout fields to manage the maximum time each request could take?
Or sometimes it becomes slower for no reason and it won't get back the same speed(rpm) as the beginning.


    rq = await RequestQueue.open(name="urls/mwm")
    from crawlee import ConcurrencySettings

    concurrency_settings = ConcurrencySettings(
        desired_concurrency=1,
        min_concurrency=1,
        max_concurrency=7,
    )
    crawler = PlaywrightCrawler(
        max_request_retries=50,
        # browser_type="firefox",
        browser_type="chromium",
        user_data_dir="./session/mwm",
        headless=True,
        request_handler=router,
        request_manager=rq,
        concurrency_settings=concurrency_settings,
        browser_launch_options={
            "args": [
                "--no-sandbox",
                "--disable-dev-shm-usage",
                "--disable-web-security",
                "--disable-extensions",
            ]
        },
        request_handler_timeout=timedelta(seconds=90),
    )
Was this page helpful?