Router for what? How it works?

I created a new crawler using the

npx crawlee create project

npx crawlee create project

command, that creates some folders and files, it creates me a router.js file, which it has an instance of

createPlaywrightRouter

createPlaywrightRouter

export const router = createPlaywrightRouter();

router.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://crawlee.dev/**'],
        label: 'detail',
    });
});

router.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });

    await Dataset.pushData({
        url: request.loadedUrl,
        title,
    });
});

export const router = createPlaywrightRouter();

router.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ['https://crawlee.dev/**'],
        label: 'detail',
    });
});

router.addHandler('detail', async ({ request, page, log }) => {
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });

    await Dataset.pushData({
        url: request.loadedUrl,
        title,
    });
});

as I understand, you are creating the default handler which is kinda the "main" listener, so later you are calling/invoking your route "detail", for the enqueLinks function, this could be interesting to split your process in more "routes"/steps, so it can be more clean and decoupled later.
My question is, how to call or invoke this without the enqueList function?
I was expecting something like:

router.addDefaultHandler(async (ctx) => {
    await ctx.invoke('extract-meta-data')
    await ctx.invoke('extract-detail')
    await ctx.invoke('download-files')
});

router.addDefaultHandler(async (ctx) => {
    await ctx.invoke('extract-meta-data')
    await ctx.invoke('extract-detail')
    await ctx.invoke('download-files')
});

Where can I see the functions this CTX admit or maybe I understood the router totally different.
Thanks

spontaneous-moccasin•4/21/23, 4:39 PM

Hi @smasher, the idea behind router is to have different routes for different types of requests depending on the label you enqueue the route with. It is pretty much equivalent of this code:

If you want to create a flow like in your code snippet, you should just in each route enqueue the next request with the next route label.

ordinary-sapphireOP•4/21/23, 4:51 PM

do you have some real example on github using this route feature with crawlee? @vojtechmaslan

Pepa J•4/24/23, 1:34 PM

@smasher To understand the CrawlerRouters you need to understand also the RequestQueue:

Let's say that RequestQueue is just queue of Request information. One of there information would be

url

url

, but also the name of the that is going to be used to process this request. The name of the is provided on the attribute.

It is not really related to the event oriented approach as you suggest. Processing of Request starts and ends always only in one single . You may of course enqueue the same request again with different (and ).

So the Router is only processing Requests from . Once the Request is successfully proceeded its lifetime ends. Once there are no Requests in the RequestQueue the Crawler ends.

You may of course build your own state machine implementation inside the similar to what @vojtechmaslan suggested, but I am not sure if it is what the original question was about.