Clear URL queue at end of run?

I'm a data reporter at CBS News using crawlee to archive web pages. Currently, when I finish a crawl, the next crawl continues to crawl pages enqueued in the previous crawl.

Is there an easy fix to this? I've looked at the docs, specifically the

persist_storage

persist_storage

and

purge_on_start

purge_on_start

parameters, but it's unclear from the documentation what exactly those do.

Happy to provide a code sample if helpful

Hall•12/4/24, 7:50 PM

Someone will reply to you shortly. In the meantime, this might help:

specific-silverOP•12/4/24, 10:09 PM

If anyone comes across this post, I think I understand what's happening now - if crawlee hits the max number of requests defined in max_requests_per_crawl, it stops making requests but doesn't clear the request queue, so if you're running enqeue you'll end up with more pages in the queue.

CChris If anyone comes across this post, I think I understand what's happening now - if...

Mantisus•12/5/24, 12:36 AM

Hi, are you by any chance using Jupiter Notebook when working with crawlee?

Since the behavior you describe corresponds to
That is, it reaches the limit and aborts, but after starting it continues where it left off, since the queue is not cleared.

But if you are working with Jupiter Notebook, the queue and cache stored in memory are not cleared without session termination.

specific-silverOP•12/5/24, 1:37 AM

Nope this is running inside a django app I'm building, not a notebook

specific-silverOP•12/5/24, 1:37 AM

I managed to get it to crawl the correct pages by not setting a limit of requests but rather limiting the crawl depth

specific-silverOP•12/5/24, 1:41 AM

However, i'm now seeing an issue where the crawler is refusing to crawl a page that it's previously crawled, not clear why

CChris However, i'm now seeing an issue where the crawler is refusing to crawl a page t...

Mantisus•12/5/24, 3:01 AM

If the crawler is to crawl the same page, you must pass

example:

specific-silverOP•12/5/24, 4:54 PM

Boom that worked for me, thanks so much for the help

MEE6•12/5/24, 4:54 PM

@Chris just advanced to level 1! Thanks for your contributions!

specific-silverOP•12/5/24, 4:56 PM

For posterity if anyone comes across this thread, I had to provide a (I used a uuid because I want them to be crawled every time) to the

Request

Request

object AND in the argument to