CheerioCrawler mixed data when using $

Hi Team!

I'm pretty new to Crawlee and I'm experimenting with the CheerioCrawlerCheerioCrawler: crawl a website and store visited URLs with their title in a database.

However, I noticed that randomly, $('title').text()$('title').text() returns the wrong data (probably from another Cheerio instance?).

I believe I'm doing something wrong since it seems to be the basic, so I'm sorry if this question has already been asked.

const crawler = new CheerioCrawler({
  requestHandler: async ({ $, enqueueLinks, request }) => {
    const { url } = request;
    const name = $("title").text();
    console.log(name, "-->", url);
    await enqueueLinks();
  },
});

await crawler.run(["https://www.example.com"]);

const crawler = new CheerioCrawler({
  requestHandler: async ({ $, enqueueLinks, request }) => {
    const { url } = request;
    const name = $("title").text();
    console.log(name, "-->", url);
    await enqueueLinks();
  },
});

await crawler.run(["https://www.example.com"]);

Would output:

1st run:

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Job positions --> https://www.example.com/internships/internship-software-development

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Job positions --> https://www.example.com/internships/internship-software-development

2nd run:

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Internship: Software Development --> https://www.example.com/internships/internship-software-development

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Internship: Software Development --> https://www.example.com/internships/internship-software-development

Apify & Crawlee•3y ago•

16 replies

nursing-lime

CheerioCrawler mixed data when using $

Hi Team!

const crawler = new CheerioCrawler({
  requestHandler: async ({ $, enqueueLinks, request }) => {
    const { url } = request;
    const name = $("title").text();
    console.log(name, "-->", url);
    await enqueueLinks();
  },
});

await crawler.run(["https://www.example.com"]);

const crawler = new CheerioCrawler({
  requestHandler: async ({ $, enqueueLinks, request }) => {
    const { url } = request;
    const name = $("title").text();
    console.log(name, "-->", url);
    await enqueueLinks();
  },
});

await crawler.run(["https://www.example.com"]);

Would output:

1st run:

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Job positions --> https://www.example.com/internships/internship-software-development

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Job positions --> https://www.example.com/internships/internship-software-development

2nd run:

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Internship: Software Development --> https://www.example.com/internships/internship-software-development

Digital Marketing Expert --> https://www.example.com/job-positions/digital-marketing-expert
Job positions --> https://www.example.com/job-positions
Internship: Software Development --> https://www.example.com/internships/internship-software-development

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,290 Members

Join

CheerioCrawler mixed data when using $

CheerioCrawler mixed data when using $

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads