clean way to stop "request queue seems to be stuck for 300.0"

A scraper that I am developing, scrapes a SPA with infinite scrolling. This works fine, but after 300 seconds, I get a

WARN

WARN

WARN

WARN , which spawns another playwright instance.
This probably happens since I only handle 1 request (I do not add anything to the RequestQueue), from which I just have a while until

finished condition

finished condition

finished condition

finished condition is met.

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

What is a clean way to stop this from happening?

Hall•1/19/25, 2:43 PM

Someone will reply to you shortly. In the meantime, this might help:

Exp•1/19/25, 2:46 PM

Hi, since the default timeout is 300 seconds (5 minutes), you can increase the timeout to allow more time for infinite scrolling.

Uunexpected-ivory Hi, since the default timeout is 300 seconds (5 minutes), you can increase the t...

moderate-tomatoOP•1/19/25, 3:15 PM

Thank you !

For others:

Actor.config.internal_timeout = timedelta(seconds=xxx)

Actor.config.internal_timeout = timedelta(seconds=xxx)

MEE6•1/19/25, 3:15 PM

@DuxSec just advanced to level 1! Thanks for your contributions!