Mmoderate-tomato
AdaptivePlaywrightCrawler starts crawling the whole web at some point.
I want to use the AdaptivePlaywrightCrawler, but it seems like it wants to crawl the entire web.
Here is my code.
Here is my code.
const crawler = new AdaptivePlaywrightCrawler({
renderingTypeDetectionRatio: 0.1,
maxRequestsPerCrawl: 50,
async requestHandler({ request, enqueueLinks, parseWithCheerio, querySelector, log, urls }) {
console.log(request.url, request.uniqueKey);
await enqueueLinks();
}
});
crawler.run(['https://crawlee.dev']);const crawler = new AdaptivePlaywrightCrawler({
renderingTypeDetectionRatio: 0.1,
maxRequestsPerCrawl: 50,
async requestHandler({ request, enqueueLinks, parseWithCheerio, querySelector, log, urls }) {
console.log(request.url, request.uniqueKey);
await enqueueLinks();
}
});
crawler.run(['https://crawlee.dev']);
