How can I override the default logs of Crawlee?

At a glance

The community member is using the basic-crawler library and Playwright, and wants to override the default logs with custom logs. They have provided a WinstonLoggerProxy class that extends the Log class and uses the winston logger to log messages. The comments provide a solution to add an internal method to the WinstonLoggerProxy class to handle different log levels. The community member also mentions that they were able to override some logs, but were still struggling to make statistics work as well. Another community member suggests handling the PERF log level differently, by treating it as DEBUG or mocking the logic to ignore those strings. There is no explicitly marked answer in the comments.

Useful resources

WWojciech

Hello I wonder how to override the default logs of crawler, this is how it looks:

This logs came from basic-crawle library: (https://github.com/apify/crawlee/blob/3ffcf56d744ac527ed8d883be3b1a62356a5930c/packages/basic-crawler/src/internals/basic-crawler.ts#L891)

I am using Playwright, and thats how I mange to override default logs with my custom like that:

Plain Text

//playwright-winston-proxy-logger.ts
import { Log } from 'crawlee'

import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger) {
    super()
    this.logger = logger
  }

  debug(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.debug(message, data)
    } else {
      this.logger.debug(message)
    }
  }

  info(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.info(message, data)
    } else {
      this.logger.info(message)
    }
  }

  warning(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.warn(message, data)
    } else {
      this.logger.warn(message)
    }
  }

  error(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, data)
    } else {
      this.logger.error(message)
    }
  }

  exception(exception: Error, message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, { exception, ...data })
    } else {
      this.logger.error(message, { exception })
    }
  }
}

and thats how I use them:

Plain Text

...
  private createCrawler = (): PlaywrightCrawler => {
    const loggerCrawler = new WinstonLoggerProxy(
      createLogger({ module: 'PLAYWRIGHT' })
    )

    return new PlaywrightCrawler({
      log: loggerCrawler, // Provide the custom logger proxy
...

Attachment

6 comments

WWojciech

Solution:

The solution is to add internal method to your proxy class, here is full code for proxy winston -> crawlee log

Plain Text

import { Log, LogLevel } from 'crawlee'

import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger) {
    super()
    this.logger = logger
  }

  debug(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.debug(message, data)
    } else {
      this.logger.debug(message)
    }
  }

  info(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.info(message, data)
    } else {
      this.logger.info(message)
    }
  }

  internal(
    level: LogLevel,
    message: string,
    data?: any,
    exception?: any
  ): void {
    switch (level) {
      case LogLevel.DEBUG:
        if (data) {
          this.logger.debug(message, { data, exception })
        } else {
          this.logger.debug(message, { exception })
        }
        break
      case LogLevel.INFO:
        if (data) {
          this.logger.info(message, { data, exception })
        } else {
          this.logger.info(message, { exception })
        }
        break
    }
  }

  warning(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.warn(message, data)
    } else {
      this.logger.warn(message)
    }
  }

  error(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, data)
    } else {
      this.logger.error(message)
    }
  }

  exception(exception: Error, message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, { exception, ...data })
    } else {
      this.logger.error(message, { exception })
    }
  }
}

WWojciech

I manage to override some logs, but still struggling to make statistic work as well

WWojciech

I manage to do this by overriding child method and passing data object to my Winston Logger, but what concern me are those 3 repeating logs

Plain Text

[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Attempting to run a task.
[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Checking for ready tasks. 
[PLAYWRIGHT] [ #1 ] [2024-08-29 13:50:00] INFO : Task will not run. No tasks are ready.

I don't have anything like that in my code, its from library:

Attempting to run a task.: https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L473
Checking for ready tasks.: https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L504
Task will not run. No tasks are ready. : https://github.com/apify/crawlee/blob/c69a34a616feda0824c88f9ec18871bff0b212c0/packages/core/src/autoscaling/autoscaled_pool.ts#L519

hi I see you are a developer at Apify, maybe you know answer to my question

WWojciech

Thats how it looks

Attachment

WWojciech

okey fixed, small issue with switch and handling PERF log level, I was treating them as INFO, and I should treat them as DEBUG or mock logic to ignore those strings:

Plain Text

import { Log, LogLevel } from 'crawlee'

import type { LoggerOptions } from 'crawlee'
import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger, options?: Partial<LoggerOptions>) {
    super(options)
    this.logger = logger
  }

  getLevel(): number {
    return super.getLevel()
  }

  setLevel(level: LogLevel): void {
    super.setLevel(level)
  }

  getOptions(): Required<LoggerOptions> {
    return super.getOptions()
  }

  setOptions(options: Partial<LoggerOptions>): void {
    super.setOptions(options)
  }

  child(options: Partial<LoggerOptions>): WinstonLoggerProxy {
    const childLogger = this.logger.child(options)

    return new WinstonLoggerProxy(childLogger, {
      ...this.getOptions(),
      ...options,
    })
  }

  internal(
    level: LogLevel,
    message: string,
    data?: any,
    exception?: any
  ): void {
    const logMethod = this.getLogMethod(level)
    if (data || exception) {
      this.logger[logMethod](message, { data, exception })
    } else {
      this.logger[logMethod](message)
    }
  }
...

  private getLogMethod(level: LogLevel): keyof Logger {
    switch (level) {
      case LogLevel.DEBUG:
      case LogLevel.PERF: //this ifxed
        return 'debug'
      case LogLevel.INFO:
        return 'info'
      case LogLevel.WARNING:
      case LogLevel.SOFT_FAIL:
        return 'warn'
      case LogLevel.ERROR:
        return 'error'
      default:
        return 'info'
    }
  }
}

AApifyBot

just advanced to level 3! Thanks for your contributions! 🎉

Add a reply

Apify Discord Mirror

How can I override the default logs of Crawlee?