RapidTech1898•3mo ago

Getting Pydantic warning when building Requeset Queue

I try to build a request queue using the following code: (the user-data i want to add is a tuple) request_queue = await Actor.open_request_queue()
for e in inpCombinations:
newReq = Request.from_url(baseLink) newReq.user_data = e await request_queue.add_request(newReq)
await Actor.exit() but i allways get this warning: C:\DEVNEU.venv\apify\Lib\site-packages\pydantic\type_adapter.py:572: UserWarning: Pydantic serializer warnings: PydanticSerializationUnexpectedValue(Expected UserData - serialized value may not be as expected [input_value=('electrician', 'Southamp...England,United Kingdom'), input_type=tuple])
return self.serializer.to_python( How can i get rid off this warning?

3 Replies

Real1ty•3mo ago

Hi! That warning comes from Pydantic’s serializer seeing that you’ve given user_data a Python tuple, even though it can technically turn it into JSON. Since user_data is expected to be a JSON-native type (like an object or array), you have a couple of clean options: Convert your tuple to a list when creating the request:

request_queue = await Actor.open_request_queue()
for e in inpCombinations:
    newReq = Request.from_url(
        baseLink,
        user_data=list(e)    # tuple → list
    )
    await request_queue.add_request(newReq)
await Actor.exit()

request_queue = await Actor.open_request_queue()
for e in inpCombinations:
    newReq = Request.from_url(
        baseLink,
        user_data=list(e)    # tuple → list
    )
    await request_queue.add_request(newReq)
await Actor.exit()

Use a dict if you want named fields:

newReq = Request.from_url(
    baseLink,
    user_data={ "role": e[0], "location": e[1], /* … */ }
)

newReq = Request.from_url(
    baseLink,
    user_data={ "role": e[0], "location": e[1], /* … */ }
)

If you really must keep tuples, you can suppress Pydantic’s warnings globally—but it’s generally better (and safer for anything consuming your queue) to stick with lists or dicts. Hope this helps!

RapidTech1898OP•2mo ago

Hello - thanks for the respone - i tried it with this code now: request_queue = await Actor.open_request_queue()
for i,e in enumerate(inpCombinations):
newReq = Request.from_url(f"{baseLink}#{i}") newReq.user_data = {"search": list(e), "excluded": inpExcludedWords} await request_queue.add_request(newReq) but i still get a pydantic error [apify] INFO [Status message]: Starting actor... C:\DEVNEU.venv\apify\Lib\site-packages\pydantic\type_adapter.py:572: UserWarning: Pydantic serializer warnings: PydanticSerializationUnexpectedValue(Expected UserData - serialized value may not be as expected [input_value={'search': ['electrician'...dom'], 'excluded': ['']}, input_type=dict]) return self.serializer.to_python( Hello - i changed the input to a list with that code

    inpCombinations = list(itertools.product(inpSearchWords, inpSearchLoc))
    inpCombinations = [list(x) for x in inpCombinations]
    Actor.log.info(inpCombinations)
    Actor.log.info(len(inpCombinations))
    
    baseLink = "https://www.bing.com/maps"
    request_queue = await Actor.open_request_queue()           
    for i,e in enumerate(inpCombinations, start=1):   
      newReq = Request.from_url(f"https://www.bing{i}.com/maps") 
      newReq.user_data = e           
      await request_queue.add_request(newReq)

    inpCombinations = list(itertools.product(inpSearchWords, inpSearchLoc))
    inpCombinations = [list(x) for x in inpCombinations]
    Actor.log.info(inpCombinations)
    Actor.log.info(len(inpCombinations))
    
    baseLink = "https://www.bing.com/maps"
    request_queue = await Actor.open_request_queue()           
    for i,e in enumerate(inpCombinations, start=1):   
      newReq = Request.from_url(f"https://www.bing{i}.com/maps") 
      newReq.user_data = e           
      await request_queue.add_request(newReq)

But now i get this Warning:

  PydanticSerializationUnexpectedValue(Expected `UserData` - serialized value may not be as expected [input_value=['builder', 'Salzburg'], input_type=list])

  PydanticSerializationUnexpectedValue(Expected `UserData` - serialized value may not be as expected [input_value=['builder', 'Salzburg'], input_type=list])

How can i get rid of that?

Real1ty•2mo ago

Hi, sorry for the late response. The problem is no longer with the user data now. Its just problem with version-mismatch warning. The code will work normally. Ensure you are using the latest version of the apify sdk as well as crawlee. If that doesn't help it, try downgrading pydantic version or just silencing the warning globally.

Getting Pydantic warning when building Requeset Queue

Did you find this page helpful?