How can I add dynamically JS string function into `preNavigationHooks`?

I would like to dynamically add a string (which describes a JS function) to preNavigationHookspreNavigationHooks array in CheerioCrawlerOptions [1]

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

If I do crawlerOptions.preNavigationHooks.push(jsFunction);crawlerOptions.preNavigationHooks.push(jsFunction);, when I run crawler, I got error:

WARN CheerioCrawler: Reclaiming failed request back to the list or queue. TypeError: hook is not a function
at CheerioCrawler._executeHooks (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\basic\internals\basic-crawler.js:834:23)
at CheerioCrawler._handleNavigation (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:326:20)
at CheerioCrawler._runRequestHandler (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:286:24)

[1] https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks

CheerioCrawlerOptions | API | Crawlee

Apify & Crawlee•4y ago•

3 replies

efficient-indigo

How can I add dynamically JS string function into `preNavigationHooks`?

I would like to dynamically add a string (which describes a JS function) to preNavigationHookspreNavigationHooks array in CheerioCrawlerOptions [1]

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

If I do crawlerOptions.preNavigationHooks.push(jsFunction);crawlerOptions.preNavigationHooks.push(jsFunction);, when I run crawler, I got error:

WARN CheerioCrawler: Reclaiming failed request back to the list or queue. TypeError: hook is not a function
at CheerioCrawler._executeHooks (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\basic\internals\basic-crawler.js:834:23)
at CheerioCrawler._handleNavigation (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:326:20)
at CheerioCrawler._runRequestHandler (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:286:24)

[1] https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks

CheerioCrawlerOptions | API | Crawlee

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,108 Members

Join

How can I add dynamically JS string function into `preNavigationHooks`?

How can I add dynamically JS string function into `preNavigationHooks`?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads