PlaywrightCrawler proxy issue

my crawler with PlaywrightCrawler works just fine but I have issue when adding proxy !!!
this is the code

import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";

const startUrls = ['http://quotes.toscrape.com/js/'];

const crawler = new PlaywrightCrawler({
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector("div.quote span.text", { "timeout": 60000 });
        const $ = await parseWithCheerio()

        const quotes = $("div.quote span.text")
        quotes.each((_, element) => { console.log($(element).text()) });
    },
});

await crawler.run(startUrls);

import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";

const startUrls = ['http://quotes.toscrape.com/js/'];

const crawler = new PlaywrightCrawler({
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector("div.quote span.text", { "timeout": 60000 });
        const $ = await parseWithCheerio()

        const quotes = $("div.quote span.text")
        quotes.each((_, element) => { console.log($(element).text()) });
    },
});

await crawler.run(startUrls);

however when I add my proxy port I always get timeout erros !!!

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ["url-to-proxy-port-im-using"]
})

// and the add it to crawler
const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  ...

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ["url-to-proxy-port-im-using"]
})

// and the add it to crawler
const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  ...

and also the same code with the proxy configuration works with CheerioCrawler !!!!
can anyone help with this issue !?

Apify & Crawlee•2y ago•

6 replies

specific-silver

PlaywrightCrawler proxy issue

my crawler with PlaywrightCrawler works just fine but I have issue when adding proxy !!!
this is the code

import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";

const startUrls = ['http://quotes.toscrape.com/js/'];

const crawler = new PlaywrightCrawler({
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector("div.quote span.text", { "timeout": 60000 });
        const $ = await parseWithCheerio()

        const quotes = $("div.quote span.text")
        quotes.each((_, element) => { console.log($(element).text()) });
    },
});

await crawler.run(startUrls);

import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";

const startUrls = ['http://quotes.toscrape.com/js/'];

const crawler = new PlaywrightCrawler({
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector("div.quote span.text", { "timeout": 60000 });
        const $ = await parseWithCheerio()

        const quotes = $("div.quote span.text")
        quotes.each((_, element) => { console.log($(element).text()) });
    },
});

await crawler.run(startUrls);

however when I add my proxy port I always get timeout erros !!!

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ["url-to-proxy-port-im-using"]
})

// and the add it to crawler
const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  ...

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: ["url-to-proxy-port-im-using"]
})

// and the add it to crawler
const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  ...

and also the same code with the proxy configuration works with CheerioCrawler !!!!
can anyone help with this issue !?

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

PlaywrightCrawler proxy issue

PlaywrightCrawler proxy issue

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads