passive-yellow•8mo ago

Customising logging

Is there a recommended way to customise logging? I want to be able to log which specific crawler and which handler a log is coming from. I have tried to override the logger in the crawler using

import defaultLog, { Log } from '@apify/log';
...
const crawler = new BasicCrawler({
  requestHandler: router,
  log: defaultLog.child({ prefix: 'MyCrawler' })
})

import defaultLog, { Log } from '@apify/log';
...
const crawler = new BasicCrawler({
  requestHandler: router,
  log: defaultLog.child({ prefix: 'MyCrawler' })
})

but then I get the following type error:

Type 'import("scrapers/node_modules/@apify/log/esm/index", { with: { "resolution-mode": "import" } }).Log' is not assignable to type 'import("scrapers/node_modules/@apify/log/cjs/index").Log'.
  Types have separate declarations of a private property 'options'.ts(2322)

Type 'import("scrapers/node_modules/@apify/log/esm/index", { with: { "resolution-mode": "import" } }).Log' is not assignable to type 'import("scrapers/node_modules/@apify/log/cjs/index").Log'.
  Types have separate declarations of a private property 'options'.ts(2322)

Thanks!

4 Replies

Hall•8mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by Matous. View answer.

luigi.ruocco•8mo ago

I think it should work if you get the log from 'apify' instead of '@apify/log'.

import { log } from 'apify';
import { BasicCrawler } from 'crawlee';

const crawler = new BasicCrawler({
    log: log.child({ prefix: 'MyCrawler' }),
    requestHandler: ({ log: myLog, request }) => {
        myLog.info(`Check this out! ${request.url}`);
    },
});

import { log } from 'apify';
import { BasicCrawler } from 'crawlee';

const crawler = new BasicCrawler({
    log: log.child({ prefix: 'MyCrawler' }),
    requestHandler: ({ log: myLog, request }) => {
        myLog.info(`Check this out! ${request.url}`);
    },
});

This works fine for me. I think there is something broken with the import path. I'll pass this to the engineering team but let me know if this works for you first.

passive-yellowOP•8mo ago

@luigi.ruocco This works for me! Had to of course install the apify package separately though which isn't ideal as I only need crawlee/logging but did the trick

NeoNomade•8mo ago

@je no need for the apify package:

import { log, LogLevel } from 'crawlee'
log.setOptions({
    logger: customLogger,
    level: LogLevel.DEBUG,
})

import { log, LogLevel } from 'crawlee'
log.setOptions({
    logger: customLogger,
    level: LogLevel.DEBUG,
})

Customising logging

Did you find this page helpful?