AdaptivePlaywrightCrawler: programmatically deciding when to render JS
AdaptivePlaywrightCrawler: programmatically deciding when to render JS
At a glance
The community member is asking if it is possible to programmatically decide when to render JavaScript using the new Adaptive Playwright Crawler. They provide an example of using HTTP crawling by default, but switching to JavaScript rendering if a certain condition is met, such as finding the word "captcha" in the loaded URL. The community member also asks how the AdaptivePlaywrightCrawler decides to render JavaScript or not.
In the comments, another community member explains that there is no official support for this functionality at the moment, as the related classes are private. However, they suggest that the community member can reuse or browse the current code to see how the prediction of rendering type works based on a custom ratio.
When asked about the possibility of adding this feature to the roadmap, a community member responds that there are no current plans, but they suggest that the community member should add a GitHub issue with the feature request. If the feature request becomes popular and attracts feedback from other users, it will then be considered.
There is no explicitly marked answer in the provided information.
Using the new adaptative Playwright crawler, is it possible programmatically decide when to render JS? For example using HTTP crawling by default, but if some condition is met (for example, finding the word 'captcha' in the loaded url), switch to JS rendering and try to unblock the page.
A similar question, for which I didn't find any answer in the docs, is how does the AdaptivePlaywrightCrawler decide to render JS or not?
Ok thanks for these precisions. Any chance of adding this to the roadmap? For example using Scrapy it's easy to mix HTTP-only and JS rendering within the same crawler, it would be great to have the same in Crawlee.
No plans, please add github issue with feature request, if it will become popular and attract feedback from other users then features will be considered.
