Mark request as handled inside response interceptor

At a glance

The community member is trying to close a page and move to the next request as soon as they have the data they need, which is available in the response to a specific request that occurs before the page is loaded. They tried to do this using preNavigationHooks, but encountered errors: "Navigation failed because browser has disconnected!" when they tried to close the page, and "requestHandler timed out after 130 seconds" when they removed the await page.close(); line.

A comment suggests the community member should try putting return in the handler or leaving it empty, but there is no explicitly marked answer.

nnathanist

All the data I need is on the response to an specific request, which occurs before the page is loaded. What I'm trying to achieve is to close the page and go to the next request as soon as I got what I need, so I tried doing it on preNavigationHooks:

preNavigationHooks: [
  async (crawlingContext, gotoOptions) => {
      const { page, request, log } = crawlingContext;
      gotoOptions.waitUntil = 'load';

      if (isProductUrl) {
        page.on('response', async (response) => {
            if (response.request().url().includes('productdetail')) {
                try {
                    const data = await response.json();
                    await Actor.pushData(data);
                    await defaultQueue.markRequestHandled(request);
                    page.removeAllListeners('response');
                    await page.close();
                } catch (err) {
                    log.error(err);
                }
            }
        });
      }
  },
]

But I'm getting this error:

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Navigation failed because browser has disconnected!

When I remove the await page.close(); line I get this error:

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. requestHandler timed out after 130 seconds (o4wrbxkzgU1eP2n).

This is the default handler. It only contains code related to enqueuing URLs:

router.addDefaultHandler(async ({ request }) => {
  if (searchPageUrlPattern.test(request.url)) {
    // Enqueue links...
  }
});

1 comment

RRado Ch.

hey! try putting return in the handler or leave it empty

Add a reply

Apify Discord Mirror

Mark request as handled inside response interceptor