Logs management - ELK
Hello, is it possible somehow to make Crawlee to output for example INFO level logs to the console and DEBUG level logs to a file ?
I want to implement the ELK stack with crawlee and looking for the best solution.
I'm open to any ideas.
Context : I have a few hundreds of crawlers written with crawlee on AWS Batch, and I want to have a dashboard where I get various stats from logs. Outputting all the logs (DEBUG) to CloudWatch would increase the cost by a lot.
3 Replies
conscious-sapphire•2y ago
I also would like to get logs from Crawlee to display on UI. Please inform me when having new ideas. Thanks.
Hi @NeoNomade , @neilnguyen.dev
There is a way how to setup a custom logger, see this example and you may play with it and introduce some cusotm formatting or file output by yourself https://gist.github.com/JJetmar/fe0e4bf3522721243d1085144021f33d
Gist
Apify Logger customization
Apify Logger customization. GitHub Gist: instantly share code, notes, and snippets.
metropolitan-bronze•14mo ago
@NeoNomade hey, did you implement this?