Crawlee Not scraping when provided with the same link twice

Hello I'm using crawlee to crawl web pages. The scene in question is I try to crawl same url multi times, I have tried add uniqueKey like await crawler.run([{url: config.url, uniqueKey: uuid}]); and

await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  }
});

await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  }
});

but it doesn't work. Then I try use requestQueue

const requestQueue = await RequestQueue.open(uuid)
await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  },
  requestQueue
});

const requestQueue = await RequestQueue.open(uuid)
await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  },
  requestQueue
});

with maxPageCrawl: 5 config, I got output

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawling: Page 1 / 5 - URL: https://www.prisma.io/docs...
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":32848,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":32848,"requestsTotal":1,"crawlerRuntimeMillis":33028}
INFO  PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
Found 1 files to combine...
data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json
Wrote 1 items to data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawling: Page 1 / 5 - URL: https://www.prisma.io/docs...
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":32848,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":32848,"requestsTotal":1,"crawlerRuntimeMillis":33028}
INFO  PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
Found 1 files to combine...
data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json
Wrote 1 items to data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json

what's the problem about my code? and how can I implement this scene?

Apify & Crawlee•3y ago•

1 reply

yielding-jade

Crawlee Not scraping when provided with the same link twice

Hello I'm using crawlee to crawl web pages. The scene in question is I try to crawl same url multi times, I have tried add uniqueKey like await crawler.run([{url: config.url, uniqueKey: uuid}]); and

await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  }
});

await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  }
});

but it doesn't work. Then I try use requestQueue

const requestQueue = await RequestQueue.open(uuid)
await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  },
  requestQueue
});

const requestQueue = await RequestQueue.open(uuid)
await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  },
  requestQueue
});

with maxPageCrawl: 5 config, I got output

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawling: Page 1 / 5 - URL: https://www.prisma.io/docs...
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":32848,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":32848,"requestsTotal":1,"crawlerRuntimeMillis":33028}
INFO  PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
Found 1 files to combine...
data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json
Wrote 1 items to data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawling: Page 1 / 5 - URL: https://www.prisma.io/docs...
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":32848,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":32848,"requestsTotal":1,"crawlerRuntimeMillis":33028}
INFO  PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
Found 1 files to combine...
data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json
Wrote 1 items to data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json

what's the problem about my code? and how can I implement this scene?

Crawlee Not scraping when provided with the same link twice

Similar Threads

Crawlee Not scraping when provided with the same link twice

Similar Threads

Similar Threads

Similar Threads