grumpy-cyan•8mo ago

clean way to stop "request queue seems to be stuck for 300.0"

A scraper that I am developing, scrapes a SPA with infinite scrolling. This works fine, but after 300 seconds, I get a WARN , which spawns another playwright instance. This probably happens since I only handle 1 request (I do not add anything to the RequestQueue), from which I just have a while until finished condition is met.

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

[crawlee.storages._request_queue] WARN  The request queue seems to be stuck for 300.0s, resetting internal state. ({"queue_head_ids_pending": 0, "in_progress": ["tEyKIytjmqjtRvA"]})

What is a clean way to stop this from happening?

4 Replies

Hall•8mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by DuxSec. View answer.

Exp•8mo ago

Hi, since the default timeout is 300 seconds (5 minutes), you can increase the timeout to allow more time for infinite scrolling.

grumpy-cyanOP•8mo ago

Thank you ! For others:

Actor.config.internal_timeout = timedelta(seconds=xxx)

Actor.config.internal_timeout = timedelta(seconds=xxx)

MEE6•8mo ago

@DuxSec just advanced to level 1! Thanks for your contributions! 🎉

clean way to stop "request queue seems to be stuck for 300.0"

Did you find this page helpful?