How do you handle crawl requests sent simultaneously by different users?

Currently, I can't crawl 2 different websites from different API calls each. The second API call is just ignored and the API continues to crawl the website of the 1st request.

I'm using Express for my API.

Alexey Udovydchenko•9/26/23, 3:56 AM

create your own random https://crawlee.dev/api/types/interface/RequestQueueHeadItem#uniqueKey

otherwise when request handled its not scraped again, its a feature to prevent duplicate requests for a same data

faint-whiteOP•9/26/23, 4:21 PM

It shows the interface but I don't see how to use it or import it

Alexey Udovydchenko•9/27/23, 5:53 AM

yes, in request object add property, generate random uuid for it by uuidv4() or something similar

faint-whiteOP•9/27/23, 1:32 PM

Is request.uniqueKey = uuidv4() what you are referring to or do I have to manually enqueue urls to add their uniqueKey?

Alexey Udovydchenko•9/28/23, 6:57 AM

yep!