How to stop following delayed javascript redirects?

moderate-tomato · 2025-03-01T15:58:15.317Z

I'm using the AdaptivePlaywrightCrawler with the same-domain strategy in enqueueLinks. The page I'm trying to crawl has delayed JavaScript redirects to other pages, such as Instagram. Sometimes, the crawler mistakenly thinks it's still on the same domain after a redirect and starts adding Instagram URLs to the main domain, like example.com/account/... and example.com/member/..., which don't actually exist, so, how can I stop following these delayed JavaScript redirects?

H

Hall•3/1/25, 3:58 PM

Someone will reply to you shortly. In the meantime, this might help:

P

Pepa J•3/3/25, 3:02 PM

Hi @Nth , Can you send us an example of how you call

enqueueLinks

enqueueLinks

enqueueLinks

enqueueLinks?

M

moderate-tomatoOP•3/3/25, 7:54 PM

Hey @Pepa J, here it's:

    router.addDefaultHandler(async (ctx) => {
      const { request, enqueueLinks, parseWithCheerio, querySelector, log, pushData, page } = ctx;
      log.info(

    router.addDefaultHandler(async (ctx) => {
      const { request, enqueueLinks, parseWithCheerio, querySelector, log, pushData, page } = ctx;
      log.info(

    router.addDefaultHandler(async (ctx) => {
      const { request, enqueueLinks, parseWithCheerio, querySelector, log, pushData, page } = ctx;
      log.info(

    router.addDefaultHandler(async (ctx) => {
      const { request, enqueueLinks, parseWithCheerio, querySelector, log, pushData, page } = ctx;
      log.info(

Running request handler for ${request.url}

);

      await enqueueLinks({
        strategy: 'same-domain',
        globs: ['http?(s)://example.com/**', 'http?(s)://**.example.com/**'],
        transformRequestFunction: (req) => {
          // Skip pdf files
          if (request.url.endsWith('.pdf')) {
            log.warning(

);

      await enqueueLinks({
        strategy: 'same-domain',
        globs: ['http?(s)://example.com/**', 'http?(s)://**.example.com/**'],
        transformRequestFunction: (req) => {
          // Skip pdf files
          if (request.url.endsWith('.pdf')) {
            log.warning(

);

      await enqueueLinks({
        strategy: 'same-domain',
        globs: ['http?(s)://example.com/**', 'http?(s)://**.example.com/**'],
        transformRequestFunction: (req) => {
          // Skip pdf files
          if (request.url.endsWith('.pdf')) {
            log.warning(

);

      await enqueueLinks({
        strategy: 'same-domain',
        globs: ['http?(s)://example.com/**', 'http?(s)://**.example.com/**'],
        transformRequestFunction: (req) => {
          // Skip pdf files
          if (request.url.endsWith('.pdf')) {
            log.warning(

* Skipping (${req.url}) - PDF

);
            return false;
          }

          return req;
        },
      });
    });

);
            return false;
          }

          return req;
        },
      });
    });

);
            return false;
          }

          return req;
        },
      });
    });

);
            return false;
          }

          return req;
        },
      });
    });

M

MEE6•3/3/25, 7:54 PM

@Nth just advanced to level 1! Thanks for your contributions!

M

moderate-tomatoOP•3/3/25, 7:58 PM

No issues with PlaywrightCrawler, but it sometimes happens with AdaptivePlaywrightCrawler

P

Pepa J•3/4/25, 8:49 AM

Thank you @Nth, I believe there might be and issue/bug that shows up happens on a specific website, would it be possible to put together minimal reproducible example with "real urls"?