sunny-green•2y ago
RequestRouter not using the `log` instance passed in its respective Playwright crawler
Hey folks, I have a bit of a weird use case here, I need to scrape multiple sites and I know the recommended approach is to use a single scraper and pass in the urls to the starting queue
But issue with that is, the logging gets messed up with all the different site logs getting mixed together, to fix that I created multiple loggers for it but the issue I am facing is,
when we import
log from crawlee it looks like its shared between all the places that has imported log too, and I have a custom logging implementation which is different for each site (mainly in the format of logs), and I need crawlee to respect that.
So to fix this I looked at the docs and createPlaywrightCrawler has an option for log where you can pass in your own logger to it, and while it uses the correct format for crawlee's internal logging, it does nothing to the logs which I explicitly log in my routehandler.
so How do I either separate log instances for each instance, or make my route handler respect the logger i have passed when I created the crawler5 Replies
Maybe You can try to get log from crawlerContext
docs:
https://crawlee.dev/api/next/playwright-crawler/interface/PlaywrightCrawlingContext#log
sunny-greenOP•2y ago
thanks! this is what I needed
sunny-greenOP•2y ago
only issue is, crawlee's internal logs like this are not respecting the log instance I passed in
sunny-greenOP•2y ago
another sample
sunny-greenOP•2y ago
nvm I just found the RequstQueue export in crawlee, thanks for the help!
I fixed this by setting a global log config in my main.ts file, and then spawning new log objects for each crawler separately