Can’t find info on url base of a crawler

how in Crawlee (Playwright) I can handle links that were already visited by the crawler? I mean, how not to repeat the links it already handled from disc memory? is

purgeRequestQueue: false

purgeRequestQueue: false

sufficient? I just don't purge the Data already done and it is handled automatically?

so for example I can crawl in chunks - 50 first urls collected from the page dynamically during the crawl run; but during second run those 50 urls would be omitted and 50 next, etc.

Saurav Jain•2/8/24, 9:46 AM

Hello @akephalos, someone from team will answer to your query soon. thanks:)

dry-scarletOP•2/8/24, 9:51 AM

thanks

lemurio•2/9/24, 7:02 PM

hey, try setting to false in the global config: https://crawlee.dev/api/core/class/Configuration#:%7E:text=CRAWLEE_PURGE_ON_START

Configuration | API | Crawlee

is a value object holding Crawlee configuration. By default, there is a
global singleton instance of this class available via .
Places that depend on a configurable behaviour depend on this class, as they have the global
instance as the default value.

Using global configuration:
```js
import { ...

dry-scarletOP•2/13/24, 5:45 PM

How the links that failed and reached can be re-visited in that way? Is this done automatically?

Aakephalos How the links that failed and reached `failedRequestHandler` can be re-visited i...

HonzaS•2/13/24, 6:50 PM

If you do not purge the requestQueue then the crawler will just continue where it stopped before. Requests that reached failedRequestHandler are considered done so they will not be revisited.

dry-scarletOP•2/13/24, 6:50 PM

how to revisit them?

HonzaS•2/13/24, 6:53 PM

you can put them again in the queue in the failedRequestHandler

HonzaS•2/13/24, 6:53 PM

under different uniqueKey of course

dry-scarletOP•2/14/24, 6:36 PM

thank you ill try it out

MEE6•2/14/24, 6:36 PM

@akephalos just advanced to level 1! Thanks for your contributions!

dry-scarletOP•2/15/24, 7:56 PM

@HonzaS can i have two different datasets, for successful and failed links? and then go back to the failed DataSet and enqeueue it again?

dry-scarletOP•2/15/24, 7:57 PM

how the failed links are meant to be handled? just forget about them? or are they revisited by some option im missing?

Aakephalos @HonzaS can i have two different datasets, for successful and failed links? and ...

HonzaS•2/15/24, 8:28 PM

You can create almost any logic so it is up to you what to do with failed links. They failed after all the retries for some reason. So for example you can put those failed requests to another named request queue and then fix the code of the crawler so they will not fail again and use this failed request queue for another run. There is no option out of the box how to revisit the failed requests that I know of.
They are considered handled because they failed many times so there is nothing the crawler can do but to skip them to continue the crawling. Otherwise there would be infinite loop and the crawler would never finish.

dry-scarletOP•2/15/24, 8:51 PM

how to reuse the request queue? i just add a link to it and it will be persisted through different runs?

dry-scarletOP•2/15/24, 8:54 PM

can i spefically pick those out? like from different dataset or else?

Oleg V.•2/19/24, 3:42 PM

This actor might me handy:
https://apify.com/lukaskrivka/rebirth-failed-requests