𝓓𝙤𝙘𝙠𝙚𝙧𝙨
𝓓𝙤𝙘𝙠𝙚𝙧𝙨•3mo ago

Crawlee PuppeteerCrawler not starting with Chrome Profile

I need a Chrome profile to run the scraper, since I need my session cookies to access precise pages. This is my code 
import { PuppeteerCrawler, Dataset } from "crawlee";
import { router } from "./routes.js";

const crawler = new PuppeteerCrawler({
    launchContext: {
        useChrome: true,
        userDataDir: 'C:\Users\enric\AppData\Local\Google\Chrome\User Data\Default',
        launchOptions: {
            headless: false,
        }
    },
    requestHandler: router,
    async failedRequestHandler({ request }) {
        // This function is called when the crawling of a request failed too many times
        await Dataset.pushData({
            url: request.url,
            succeeded: false,
            errors: request.errorMessages,
        })
    },
});

await crawler.run([
    'mylink'
]);
import { PuppeteerCrawler, Dataset } from "crawlee";
import { router } from "./routes.js";

const crawler = new PuppeteerCrawler({
    launchContext: {
        useChrome: true,
        userDataDir: 'C:\Users\enric\AppData\Local\Google\Chrome\User Data\Default',
        launchOptions: {
            headless: false,
        }
    },
    requestHandler: router,
    async failedRequestHandler({ request }) {
        // This function is called when the crawling of a request failed too many times
        await Dataset.pushData({
            url: request.url,
            succeeded: false,
            errors: request.errorMessages,
        })
    },
});

await crawler.run([
    'mylink'
]);
Still the crawler opens without the Chrome Profile 🙂
1 Reply
Oleg V.
Oleg V.•3mo ago
Maybe You can use preNavigationHooks to add extra cookies: https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#preNavigationHooks Or modify Your session using createSessionFunction with your cookies: https://crawlee.dev/js/api/next/core/interface/SessionPoolOptions#createSessionFunction

Did you find this page helpful?