knufri•3w ago

My actors suddenly throw errors according to broken head in request queue

Hello together! I have a strange problem, my different actors working fine for months but suddenly they throwing errors like this:

[apify.scrapy._async_thread] ERROR Coroutine execution raised an exception.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/apify/scrapy/_async_thread.py", line 62, in run_coro
return future.result(timeout=timeout.total_seconds())
[...]
File "/usr/local/lib/python3.12/site-packages/crawlee/storages/_request_queue.py", line 232, in fetch_next_request
return await self._client.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_client.py", line 100, in fetch_next_request
return await self._implementation.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 205, in fetch_next_request
await self._ensure_head_is_non_empty()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 221, in _ensure_head_is_non_empty
await self._list_head()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 250, in _list_head
request = Request.model_validate(
File "/usr/local/lib/python3.12/site-packages/pydantic/main.py", line 705, in model_validate
return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for Request
Input should be a valid dictionary or instance of Request [type=model_type, input_value=None, input_type=NoneType]

[apify.scrapy._async_thread] ERROR Coroutine execution raised an exception.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/apify/scrapy/_async_thread.py", line 62, in run_coro
return future.result(timeout=timeout.total_seconds())
[...]
File "/usr/local/lib/python3.12/site-packages/crawlee/storages/_request_queue.py", line 232, in fetch_next_request
return await self._client.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_client.py", line 100, in fetch_next_request
return await self._implementation.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 205, in fetch_next_request
await self._ensure_head_is_non_empty()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 221, in _ensure_head_is_non_empty
await self._list_head()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 250, in _list_head
request = Request.model_validate(
File "/usr/local/lib/python3.12/site-packages/pydantic/main.py", line 705, in model_validate
return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for Request
Input should be a valid dictionary or instance of Request [type=model_type, input_value=None, input_type=NoneType]

according to my research this is caused by an empty or broken request queue. Sometimes caused by very long url with parameters and a broken unique id for the queue. But I don't use any long url and I use the apify request qeue like in the docs. If I run the actor on my local maschine all works fine. I spent days now to debug and find a solution so any help is welcome!

message.txt

3 Replies

knufriOP•3w ago

If somebody reads this post later on, this was caused by bugs in crawlee und apify which fixed in the actual versions:

apify==3.0.3
crawlee==1.0.4

apify==3.0.3
crawlee==1.0.4

knufriOP•3w ago

https://github.com/apify/apify-sdk-python/issues/630

GitHub

Apify RQ client list_head method returns truncated unique keys an...

Description When calling ApifyRequestQueueSingleClient._list_head(), the response contains truncated values of unique_key and url for long URLs. Longer than 128 chars, see https://github.com/apify/...

knufriOP•3w ago

https://github.com/apify/crawlee-python/issues/1472

GitHub

Pydantic validation error in request queue after request retries ·...

Hello Apify devs! This week, my scraper which has been using crawlee[parsel] == 0.5.1 and apify==2.2.0 stopped working (returning 0 results, logs indicate the requests somehow getting blocked but I...

My actors suddenly throw errors according to broken head in request queue

Did you find this page helpful?