Apify Discord Mirror

Updated 2 years ago

How to prevent following redirects to other domains?

At a glance

The community member is looking for a way to set up request interception globally in a crawler, rather than having to do it on each page load. They mention using await page.setRequestInterception(true) and handling requests individually, but want a more global solution. The comments suggest using preNavigationHooks, which the community member was able to find and use to solve their issue, although they didn't end up blocking the requests in the end.

MMichal

I see there is a way to prevent this once page loads with something like this:

Plain Text

    await page.setRequestInterception(true);

    page.on('request', async (request) => {
      if (ok) {
        await request.continue();
      } else {
        await request.abort();
      }
    }

But how about when I have a url in queue, is there a nice way to have the request interception setup in the crawler options instead and have it defined globally?

4 comments

MMichal

I guess I'm looking for something like a global beforeNavigation callback.

MMichal

Yay, it's preNavigationHooks, I wasn't able to find it in the API docs though. Found it here in fulltext and then used github all code search for examples.

PPepa J

Yes, you need to put this to the prenavigationHooks, did this solve your issue? 🙂

MMichal

Yep I got it working, although I didn't end up blocking them in the end.

Add a reply