knufriK
Apify & Crawlee•2mo ago
knufri

My actors suddenly throw errors according to broken head in request queue

Hello together! I have a strange problem, my different actors working fine for months but suddenly they throwing errors like this:
[apify.scrapy._async_thread] ERROR Coroutine execution raised an exception.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/apify/scrapy/_async_thread.py", line 62, in run_coro
return future.result(timeout=timeout.total_seconds())
[...]
File "/usr/local/lib/python3.12/site-packages/crawlee/storages/_request_queue.py", line 232, in fetch_next_request
return await self._client.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_client.py", line 100, in fetch_next_request
return await self._implementation.fetch_next_request()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 205, in fetch_next_request
await self._ensure_head_is_non_empty()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 221, in _ensure_head_is_non_empty
await self._list_head()
File "/usr/local/lib/python3.12/site-packages/apify/storage_clients/_apify/_request_queue_single_client.py", line 250, in _list_head
request = Request.model_validate(
File "/usr/local/lib/python3.12/site-packages/pydantic/main.py", line 705, in model_validate
return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for Request
Input should be a valid dictionary or instance of Request [type=model_type, input_value=None, input_type=NoneType]

according to my research this is caused by an empty or broken request queue. Sometimes caused by very long url with parameters and a broken unique id for the queue. But I don't use any long url and I use the apify request qeue like in the docs. If I run the actor on my local maschine all works fine. I spent days now to debug and find a solution so any help is welcome!
message.txt5.99KB
Solution
If somebody reads this post later on, this was caused by bugs in crawlee und apify which fixed in the actual versions:
apify==3.0.3
crawlee==1.0.4
Jump to solution
Was this page helpful?