PlaywrightCrawler New Instance unexpected result

Hi guys, I'm new to crawlee.

I wrap the sample code into a function.
Each time the getAvailableURLsgetAvailableURLs function is called, a new instance of the PlaywrightCrawlerPlaywrightCrawler class is created and used to crawl the provided URL.

Source Code:

const getAvailableURLs = async (url: string, maxRequestsPerCrawl: number, maxRequestRetries: number, strategy: "all" | "same-hostname" | "same-domain" | "same-origin") => {
    let availableUrls: string[] = [];
    const crawler = new PlaywrightCrawler({
        maxRequestsPerCrawl,
        maxRequestRetries,
        // Use the requestHandler to process each of the crawled pages.
        async requestHandler({ request, enqueueLinks }) {

            await availableUrls.push(request.loadedUrl ? request.loadedUrl : "")


            await enqueueLinks(
                {
                    strategy
                }
            );
        },
    });

    await crawler.run([url])

    return availableUrls
}


let availableUrls = await getAvailableURLs('https://crawlee.dev/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)


availableUrls = await getAvailableURLs('https://cheerio.js.org/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)

const getAvailableURLs = async (url: string, maxRequestsPerCrawl: number, maxRequestRetries: number, strategy: "all" | "same-hostname" | "same-domain" | "same-origin") => {
    let availableUrls: string[] = [];
    const crawler = new PlaywrightCrawler({
        maxRequestsPerCrawl,
        maxRequestRetries,
        // Use the requestHandler to process each of the crawled pages.
        async requestHandler({ request, enqueueLinks }) {

            await availableUrls.push(request.loadedUrl ? request.loadedUrl : "")


            await enqueueLinks(
                {
                    strategy
                }
            );
        },
    });

    await crawler.run([url])

    return availableUrls
}


let availableUrls = await getAvailableURLs('https://crawlee.dev/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)


availableUrls = await getAvailableURLs('https://cheerio.js.org/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)

Result:
1st Crawl: INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.
2nd Crawl: INFO PlaywrightCrawler: Terminal status message: Finished! Total 0 requests: 0 succeeded, 0 failed.INFO PlaywrightCrawler: Terminal status message: Finished! Total 0 requests: 0 succeeded, 0 failed.

Expected Result:
1st Crawl: INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.
2nd Crawl: INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.INFO PlaywrightCrawler: Terminal status message: Finished! Total 3 requests: 3 succeeded, 0 failed.

Question
How do I achieve expected resultexpected result and able to customize strategystrategy, maxRequestsPerCrawlmaxRequestsPerCrawl and maxRequestRetriesmaxRequestRetries by passing in parameters?

Apify & Crawlee•4y ago•

2 replies

hurt-tomato

PlaywrightCrawler New Instance unexpected result

const getAvailableURLs = async (url: string, maxRequestsPerCrawl: number, maxRequestRetries: number, strategy: "all" | "same-hostname" | "same-domain" | "same-origin") => {
    let availableUrls: string[] = [];
    const crawler = new PlaywrightCrawler({
        maxRequestsPerCrawl,
        maxRequestRetries,
        // Use the requestHandler to process each of the crawled pages.
        async requestHandler({ request, enqueueLinks }) {

            await availableUrls.push(request.loadedUrl ? request.loadedUrl : "")


            await enqueueLinks(
                {
                    strategy
                }
            );
        },
    });

    await crawler.run([url])

    return availableUrls
}


let availableUrls = await getAvailableURLs('https://crawlee.dev/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)


availableUrls = await getAvailableURLs('https://cheerio.js.org/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)

const getAvailableURLs = async (url: string, maxRequestsPerCrawl: number, maxRequestRetries: number, strategy: "all" | "same-hostname" | "same-domain" | "same-origin") => {
    let availableUrls: string[] = [];
    const crawler = new PlaywrightCrawler({
        maxRequestsPerCrawl,
        maxRequestRetries,
        // Use the requestHandler to process each of the crawled pages.
        async requestHandler({ request, enqueueLinks }) {

            await availableUrls.push(request.loadedUrl ? request.loadedUrl : "")


            await enqueueLinks(
                {
                    strategy
                }
            );
        },
    });

    await crawler.run([url])

    return availableUrls
}


let availableUrls = await getAvailableURLs('https://crawlee.dev/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)


availableUrls = await getAvailableURLs('https://cheerio.js.org/', 2, 1, "same-hostname")
console.info(`availableUrls length: ${availableUrls.length} result: ${availableUrls}`)

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,508 Members

Join

PlaywrightCrawler New Instance unexpected result

PlaywrightCrawler New Instance unexpected result

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads