How to reset a queue?

At a glance

The community member is developing a scraper and facing an issue where they want to re-run the scraper on the same page, but calling drop() on the request queue fails with an error that the queue does not exist. The community members discuss that when using a named request queue, which is meant to be persistent, they should be able to drop it and create a new one with the same name for development. They suggest trying to re-open the queue after dropping it, and using await to fix the issue. The community members confirm that dropping the queue and re-opening it with the same name is the right way to get a new empty queue.

MMichal

While developing a scraper I'm often facing this issue:
1) I add initial page to the queue
2) I run the scraper, which marks the url done
3) I want to re-run the scraper on the same page

I know I can keep renaming the queue name, but is there a way to reset/clear the queue instead?

if I call drop() on it, it simply fails with:

Plain Text

Request queue with id: 7ae80a2d-3b06-4a8f-929d-4fbfc5947e81 does not exist.

14 comments

PPepa J

Hello ,
Since you are using named RequestQueue (which is meant to be persistent) you should be able drop it on start and (probably the importatnt part there) create a new one with the same name again for the development.

You may also use unnamed (default) RequestQueue, and then if you run scraper locally with apify-cli via apify run -p the default RequestQueue would be deleted for you, everytime you run it. Be aware that you may have only single unnamed (default) RequestQueue in your run.

MMichal

Since you are using named RequestQueue (which is meant to be persistent) you should be able drop it on start and (probably the importatnt part there) create a new one with the same name again for the development.

Thanks, but how?

AApifyBot

just advanced to level 2! Thanks for your contributions! 🎉

MMichal

I am trying to remove it but this fails right away with the above error:

Plain Text

      options.requestQueue = await RequestQueue.open(`${rqName}_V${this.version}`);
      if (this.args.clear) {
        options.requestQueue.drop();
      }

MMichal

this is done at the very beginning

MMichal

maybe I changed the name of the queue already and it fails because it dodes not technically exist yet.. will do some testing

MMichal

or should it be written more like this, with re-opening the queue?

Plain Text

      options.requestQueue = await RequestQueue.open(nameWithVersion);

      if (this.args.clear) {
        options.requestQueue.drop();
        options.requestQueue = await RequestQueue.open(nameWithVersion);
      }

MMichal

in either case it keeps failing if I call drop()

HHonzaS

isn't drop also async? there should be await
I am not sure it will solve your problem though

MMichal

can you at least tell me if I should re-open it or not?

MMichal

await did fix the issue - thanks!

MMichal

and I'm reopening it - is that the right way?

HHonzaS

yes, droping it and reopening the the queue with the same name will give you new empty queue

btw. if await fixed the issue then it did not failed before on drop(), no? that is not making sense

MMichal

Correct, I misread the exception trace, it was failing when the crawler was initiated.

Add a reply

Apify Discord Mirror

How to reset a queue?