how to set payload in cheerio crawler preNavigationHook...

At a glance

The community member is having an issue with their code, where setting the request.payload property does not work and results in a ReferenceError: page is not defined error. They also have a question about whether to use gotOptions.headers or request.headers to set headers.

In the comments, another community member suggests that this might be a bug and that they will inform the team. Another community member requests to see the full error stack, as under Cheerio, the page variable should not be used. They also guess that the issue might be related to setting the payload for a GET request, causing the crawler to go wild.

A third community member confirms that they use request.headers in one of their actors, and it works fine.

The original community member then realizes that the issue was a bug in their own code, where they had a long payload that was out of the screen, causing them to miss the issue.

There is no explicitly marked answer in the comments.

HHonzaS

doing it like this:

Plain Text

preNavigationHooks:[async (crawlingContext, gotOptions) => {
         const { request } = crawlingContext;
             request.payload = `.......`;
 }

does not work, error: ReferenceError: page is not defined
also when I want to set headers, should I use gotOptions.headers= or request.headers=
what is the difference?

4 comments

LLukas Krivka

Looks like a bug, will tell team to check

AAlexey Udovydchenko

Need to see full error stack since under Cheerio page not supposed to be used. I guess you might be setting payload for GET request and crawler gone wild.

AAlexey Udovydchenko

As for headers - I using request.headers in one of the actors, works fine

HHonzaS

so it was bug in my code 🤦‍♂️ it has nothing to do with puppeteer page
I had payload like this .....${page} but it was too long and way out of the screen to notice it

Add a reply

Apify Discord Mirror

how to set payload in cheerio crawler preNavigationHooks