rubber-blueR
Apify & Crawlee2y ago
7 replies
rubber-blue

How can I override the default logs of Crawlee?

Hello I wonder how to override the default logs of crawler, this is how it looks:


This logs came from basic-crawle library: (https://github.com/apify/crawlee/blob/3ffcf56d744ac527ed8d883be3b1a62356a5930c/packages/basic-crawler/src/internals/basic-crawler.ts#L891)

I am using Playwright, and thats how I mange to override default logs with my custom like that:

//playwright-winston-proxy-logger.ts
import { Log } from 'crawlee'

import type { Logger } from 'winston'

type AdditionalData = Record<string, unknown> | null

export class WinstonLoggerProxy extends Log {
  private logger: Logger

  constructor(logger: Logger) {
    super()
    this.logger = logger
  }

  debug(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.debug(message, data)
    } else {
      this.logger.debug(message)
    }
  }

  info(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.info(message, data)
    } else {
      this.logger.info(message)
    }
  }

  warning(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.warn(message, data)
    } else {
      this.logger.warn(message)
    }
  }

  error(message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, data)
    } else {
      this.logger.error(message)
    }
  }

  exception(exception: Error, message: string, data?: AdditionalData): void {
    if (data) {
      this.logger.error(message, { exception, ...data })
    } else {
      this.logger.error(message, { exception })
    }
  }
}


and thats how I use them:

...
  private createCrawler = (): PlaywrightCrawler => {
    const loggerCrawler = new WinstonLoggerProxy(
      createLogger({ module: 'PLAYWRIGHT' })
    )

    return new PlaywrightCrawler({
      log: loggerCrawler, // Provide the custom logger proxy
...
image.png
GitHub
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, an...
crawlee/packages/basic-crawler/src/internals/basic-crawler.ts at 3f...
Was this page helpful?