Pprogressive-amaranth
Crawlee scrapper invoking the same handler multiple times
Hey all! I've built a Crawlee scrapper, but for some reason it invokes the same handler multiple times, creating a lot of duplicate requests and entries in my dataset. Also:
- I've already tried manually setting uniqueKeys for all my requests.
- I've also tried setting maxConcurrency: 1 for the crawler.
- As you can see from the logs below, the issue is not that I'm adding the same requests multiple times. It's Crawlee who's invoking handlers multiple times with the same request.
Has anyone experienced the same issue? Any clue about what could be happening here?
I've posted the question and all the details (code and logs) on StackOverflow: https://stackoverflow.com/questions/77358550/crawlee-scrapper-visiting-the-same-url-multiple-times
- I've already tried manually setting uniqueKeys for all my requests.
- I've also tried setting maxConcurrency: 1 for the crawler.
- As you can see from the logs below, the issue is not that I'm adding the same requests multiple times. It's Crawlee who's invoking handlers multiple times with the same request.
Has anyone experienced the same issue? Any clue about what could be happening here?
I've posted the question and all the details (code and logs) on StackOverflow: https://stackoverflow.com/questions/77358550/crawlee-scrapper-visiting-the-same-url-multiple-times
Stack Overflow
I've built a Crawlee scrapper, but for some reason it invokes the same handler multiple times, creating a lot of duplicate requests and entries in my dataset. Also:
I've already tried manually set...
I've already tried manually set...
