Loadtime

Hello,

Is there a way to get the load time of a site from crawlee in headless mode? I'm using PlaywrightCrawler.

Thanks!

Hall•3/1/25, 11:08 PM

Someone will reply to you shortly. In the meantime, this might help:

Pepa J•3/3/25, 2:52 PM

Hi @BOBPG :

Do you have something like this in your mind?

instant-harlequinOP•3/3/25, 9:24 PM

Someting like this looks great, i'm going to try this, thanks!

const crawler = new PlaywrightCrawler({
    // ...
    preNavigationHooks: [async ({ request }, gotoOptions) => {
        requestStartTimes.set(request.id!, Date.now());
        gotoOptions!.waitUntil = 'load';
    }],
    requestHandler: async ({ request, log }) => {
        const loadingTime = Date.now() - requestStartTimes.get(request.id!)!;
        log.info(`Loading ${request.url} took ${loadingTime / 1000}s`);
    },
});

const crawler = new PlaywrightCrawler({
    // ...
    preNavigationHooks: [async ({ request }, gotoOptions) => {
        requestStartTimes.set(request.id!, Date.now());
        gotoOptions!.waitUntil = 'load';
    }],
    requestHandler: async ({ request, log }) => {
        const loadingTime = Date.now() - requestStartTimes.get(request.id!)!;
        log.info(`Loading ${request.url} took ${loadingTime / 1000}s`);
    },
});