evident-indigo
evident-indigo3y ago

Use page.on('request') in PuppeteerCrawler

How can i do that with the PuppeteerCrawler?
page.on('request', async (request) => {
// check if request.url contains any of the google domains
googleDomains.forEach((domain) => {
// if domain is found in the request.url push the url to the foundObjects.foundDomains array
if (request.url().includes(domain)) {
// show spinner info
found.foundDomains.push(request.url());
takeScreenshot = true;
}
});
});
page.on('request', async (request) => {
// check if request.url contains any of the google domains
googleDomains.forEach((domain) => {
// if domain is found in the request.url push the url to the foundObjects.foundDomains array
if (request.url().includes(domain)) {
// show spinner info
found.foundDomains.push(request.url());
takeScreenshot = true;
}
});
});
3 Replies
HonzaS
HonzaS3y ago
You need to put this code to the preNavigationHooks in crawlers constructor.
evident-indigo
evident-indigoOP3y ago
Works - thanks for the hint @HonzaS is the current url available in the preNavigationHooks or how could i log the my result
HonzaS
HonzaS3y ago
shuld be in request in crawlingContext

Did you find this page helpful?