ambitious-aqua•3y ago
requestQueue doesn't delete requests after visiting and saving data
Hi, working with crawlee and playwright. I've noticed that requests aren't being popped out of the queue even though the links have already been visited and scraped. Am I missing a configuration or something?
10 Replies
ambitious-aquaOP•3y ago
The queue looks like this even after all of these requests have already been visited:
ambitious-aquaOP•3y ago
ambitious-aquaOP•3y ago
my default router ( I have one other router for the DETAILS request but it does not enqueue links) :
ambitious-aquaOP•3y ago
ambitious-aquaOP•3y ago
it also seems like it dumps the requests back to the queue after scraping...
ambitious-aquaOP•3y ago
ambitious-aquaOP•3y ago
follow up: I'm dumb and in one case there wasn't a <p> element so the selector wasn't finding anything which is why it was failing.
@tegra just advanced to level 1! Thanks for your contributions! 🎉
Its a feature, requests stored per-run to process unique URL just once, this way you can add all sublinks from web site without going into endless loop of scraping
ambitious-aquaOP•3y ago
thanks, it took me awhile but I read the documentation and realized that was the case!