Apify

Apify Discord Mirror

Updated 2 weeks ago

Double log output

Apify platform
At a glance

The community member is experiencing an issue where logging is printed twice in routes.py, while it works as expected in main.py. They tried setting up logging in main.py but that caused everything to be printed twice in main.py and three times in routes.py.

The community members suggested setting up logging in a separate file and importing it everywhere, but that did not resolve the issue. After further investigation, they found that the problem was that the setup_logging() function was being called multiple times in routes.py, leading to duplicate log handlers and duplicated logs.

The solution provided by the community members was to add the following code in utils.py:

logging.getLogger().handlers.clear() apify_logger = logging.getLogger("apify") apify_logger.setLevel(logging.DEBUG)

This fixed the issue and the community member expressed their gratitude.

Useful resources
DDuxSec
·
in main.py logging works as expected, however in routes.py logging is printed twice for some reason.
I did not setup any custom logging, I just use
Actor.log.info("STARTING A NEW CRAWL JOB")

example:
Plain Text
[apify] INFO  Checking item 17
[apify] INFO  Checking item 17 ({"message": "Checking item 17"})
[apify] INFO  Processing new item with index: 17
[apify] INFO  Processing new item with index: 17 ({"message": "Processing new item with index: 17"})


If I add this in my main.py (https://docs.apify.com/sdk/python/docs/concepts/logging)
Plain Text
async def main() -> None:
    async with Actor:
        ##### SETUP LOGGING #####
        handler = logging.StreamHandler()
        handler.setFormatter(ActorLogFormatter())

        apify_logger = logging.getLogger('apify')
        apify_logger.setLevel(logging.DEBUG)
        apify_logger.addHandler(handler)

it prints everything from main.py 2x, and everything from routes.py 3x.

Plain Text
[apify] INFO  STARTING A NEW CRAWL JOB
[apify] INFO  STARTING A NEW CRAWL JOB ({"message": "STARTING A NEW CRAWL JOB"})
[apify] INFO  STARTING A NEW CRAWL JOB ({"message": "STARTING A NEW CRAWL JOB"})
EExp
Marked as solution
Add following code in utils.py
logging.getLogger().handlers.clear() apify_logger = logging.getLogger("apify") apify_logger.setLevel(logging.DEBUG)
View full solution
E
D
11 comments
EExp
Hi, modify your logging setup in main.py
async def main() -> None: async with Actor: apify_logger = logging.getLogger('apify') if not apify_logger.hasHandlers(): handler = logging.StreamHandler() handler.setFormatter(ActorLogFormatter()) apify_logger.setLevel(logging.DEBUG) apify_logger.addHandler(handler)
DDuxSec
Thanks for the reply, if I use your code, it still logs everything in routes.py 2x... (it works in main.py correctly)
EExp
If you set up logging in a separate file (logging_setup.py), then import it everywhere, it ensures consistency.
DDuxSec
still the same... main printed OK, but routes printed 2x

main.py
Plain Text
async def main() -> None:
    async with Actor:
        ##### SETUP LOGGING #####
        await setup_logging()
        ...


utils.py
Plain Text
from apify.log import ActorLogFormatter
import logging

async def setup_logging():
    apify_logger = logging.getLogger('apify')
    if not apify_logger.hasHandlers():
        handler = logging.StreamHandler()
        handler.setFormatter(ActorLogFormatter())

        apify_logger.setLevel(logging.DEBUG)
        apify_logger.addHandler(handler)


routes.py
Plain Text
@router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
    await setup_logging()

    Actor.log.info("STARTING A NEW CRAWL JOB")
EExp
The issue is that each time default_handler is called in routes.py, it re-adds a new log handler. Since setup_logging() runs inside default_handler, new handlers keep stacking up, leading to duplicated logs.
EExp
fix utils.py
from apify.log import ActorLogFormatter import logging LOGGING_INITIALIZED = False async def setup_logging(): global LOGGING_INITIALIZED if LOGGING_INITIALIZED: return apify_logger = logging.getLogger('apify') if not apify_logger.hasHandlers(): handler = logging.StreamHandler() handler.setFormatter(ActorLogFormatter()) apify_logger.setLevel(logging.DEBUG) apify_logger.addHandler(handler) LOGGING_INITIALIZED = True
EExp
Fix routes.py
from utils import setup_logging from apify import Actor @router.default_handler async def default_handler(context: PlaywrightCrawlingContext) -> None: await setup_logging() Actor.log.info("STARTING A NEW CRAWL JOB")
DDuxSec
ah I see, I implemented your code, but still same issue... (routes.py does see LOGGING_INITIALIZED correctly and returns early)
----
I created a sample project with the minimal code to reproduce the issue

main.py
Plain Text
from apify import Actor
from crawlee.crawlers import PlaywrightCrawler
from src.routes import router
from src.utils import setup_logging

async def main() -> None:
    async with Actor:
        await setup_logging()
        crawler = PlaywrightCrawler(headless=False, request_handler=router)
        await crawler.run(["https://apify.com"])


routes.py
Plain Text
from crawlee.crawlers import PlaywrightCrawlingContext
from crawlee.router import Router
from apify import Actor
from src.utils import setup_logging

router = Router[PlaywrightCrawlingContext]()

@router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
    await setup_logging()  
    Actor.log.info("STARTING A NEW CRAWL JOB")


utils.py
Plain Text
from apify.log import ActorLogFormatter
import logging

LOGGING_INITIALIZED = False  

async def setup_logging():
    global LOGGING_INITIALIZED
    if LOGGING_INITIALIZED:
        return 

    apify_logger = logging.getLogger('apify')

    if not apify_logger.hasHandlers():
        handler = logging.StreamHandler()
        handler.setFormatter(ActorLogFormatter())

        apify_logger.setLevel(logging.DEBUG)
        apify_logger.addHandler(handler)

    LOGGING_INITIALIZED = True
EExp
Add following code in utils.py
logging.getLogger().handlers.clear() apify_logger = logging.getLogger("apify") apify_logger.setLevel(logging.DEBUG)
DDuxSec
that fixed it! thank you so much
EExp
I am glad it's resolved
Add a reply
Sign up and join the conversation on Discord