4 replies

Unexpected results when currying a value into the request handler

I need to curry a value into my request handler. When I do this, my first invocation of the crawler runs as expected, but the second invocation reaches maxRequestsPerCrawl immediately (even when passing in a new url).
I see this on the first run:

...
INFO  PlaywrightCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 2 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 5 requests and will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":5,"requestsFailed":0,"retryHistogram":[5],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":13368,"requestsFinishedPerMinute":9,"requestsFailedPerMinute":0,"requestTotalDurationMillis":66838,"requestsTotal":5,"crawlerRuntimeMillis":32684}
INFO  PlaywrightCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed. {"terminal":true}

...
INFO  PlaywrightCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 2 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 5 requests and will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":5,"requestsFailed":0,"retryHistogram":[5],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":13368,"requestsFinishedPerMinute":9,"requestsFailedPerMinute":0,"requestTotalDurationMillis":66838,"requestsTotal":5,"crawlerRuntimeMillis":32684}
INFO  PlaywrightCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed. {"terminal":true}

Then this on the second:

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawler reached the maxRequestsPerCrawl limit of 2 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":0,...

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawler reached the maxRequestsPerCrawl limit of 2 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":0,...

I have a function that takes a single string value (

step_id

step_id

) and passes that value into a function that returns the request handler.

export function crawler<T>(step_id: string) {
  const playwright_crawler = new PlaywrightCrawler({
    maxRequestsPerCrawl: 2,
    requestHandler: create_crawlee_rec_handler(step_id),
  });
  return playwright_crawler;
}

export function crawler<T>(step_id: string) {
  const playwright_crawler = new PlaywrightCrawler({
    maxRequestsPerCrawl: 2,
    requestHandler: create_crawlee_rec_handler(step_id),
  });
  return playwright_crawler;
}

This function is imported and called as such:

const crawlerInstance = crawler(`${step.id}`);
    await crawlerInstance.run([href]);

const crawlerInstance = crawler(`${step.id}`);
    await crawlerInstance.run([href]);

If I don't curry the step_id into the request handler, the second run works just fine. Maybe there is another way to get my step_id value into the scope of the request handler?
Thanks in advance for any advice.

Unexpected results when currying a value into the request handler

...
INFO  PlaywrightCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 2 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 5 requests and will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":5,"requestsFailed":0,"retryHistogram":[5],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":13368,"requestsFinishedPerMinute":9,"requestsFailedPerMinute":0,"requestTotalDurationMillis":66838,"requestsTotal":5,"crawlerRuntimeMillis":32684}
INFO  PlaywrightCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed. {"terminal":true}

...
INFO  PlaywrightCrawler: Earlier, the crawler reached the maxRequestsPerCrawl limit of 2 requests and all requests that were in progress at that time have now finished. In total, the crawler processed 5 requests and will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":5,"requestsFailed":0,"retryHistogram":[5],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":13368,"requestsFinishedPerMinute":9,"requestsFailedPerMinute":0,"requestTotalDurationMillis":66838,"requestsTotal":5,"crawlerRuntimeMillis":32684}
INFO  PlaywrightCrawler: Finished! Total 5 requests: 5 succeeded, 0 failed. {"terminal":true}

Then this on the second:

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawler reached the maxRequestsPerCrawl limit of 2 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":0,...

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawler reached the maxRequestsPerCrawl limit of 2 requests and will shut down soon. Requests that are in progress will be allowed to finish.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":0,...

I have a function that takes a single string value (

step_id

step_id

) and passes that value into a function that returns the request handler.

export function crawler<T>(step_id: string) {
  const playwright_crawler = new PlaywrightCrawler({
    maxRequestsPerCrawl: 2,
    requestHandler: create_crawlee_rec_handler(step_id),
  });
  return playwright_crawler;
}

export function crawler<T>(step_id: string) {
  const playwright_crawler = new PlaywrightCrawler({
    maxRequestsPerCrawl: 2,
    requestHandler: create_crawlee_rec_handler(step_id),
  });
  return playwright_crawler;
}

This function is imported and called as such:

const crawlerInstance = crawler(`${step.id}`);
    await crawlerInstance.run([href]);

const crawlerInstance = crawler(`${step.id}`);
    await crawlerInstance.run([href]);

Unexpected results when currying a value into the request handler

Unexpected results when currying a value into the request handler

Similar Threads

Similar Threads

Similar Threads