stormy-gold•5mo ago

Camoufox and adaptive playwright

Hello great friends of Crawlee, I was wondering if there was anyway to use camoufox and the adaptive playwright browser? It seems to throw an error when I try to add the browser pool.

6 Replies

Hall•5mo ago

Someone will reply to you shortly. In the meantime, this might help:

memo23•5mo ago

hey @Hall which site you want to scrape?

Mantisus•5mo ago

Hey, @Doigus Could you create an Issue, with an example of the error you're getting and more context? https://github.com/apify/crawlee-python/issues

GitHub

Issues · apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Wo...

stormy-goldOP•5mo ago

More so there is seemingly no option to add brower_pool as with Playwright

Mantisus•5mo ago

The brower_pool is set with playwright_crawler_specific_kwargs, but I don't have a way to test running it with Camoufox right now. However, if it is not supported. it is an error

async def main() -> None:
    crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
        max_requests_per_crawl=10,
        playwright_crawler_specific_kwargs={'browser_pool': BrowserPool(plugins=[
            PlaywrightBrowserPlugin(browser_type='chromium')
            ])}
    )

    @crawler.router.default_handler
    async def default_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

    await crawler.run(['https://crawlee.dev/'])

async def main() -> None:
    crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
        max_requests_per_crawl=10,
        playwright_crawler_specific_kwargs={'browser_pool': BrowserPool(plugins=[
            PlaywrightBrowserPlugin(browser_type='chromium')
            ])}
    )

    @crawler.router.default_handler
    async def default_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

    await crawler.run(['https://crawlee.dev/'])

MEE6•5mo ago

@Doigus just advanced to level 1! Thanks for your contributions! 🎉

Camoufox and adaptive playwright

Did you find this page helpful?