custom logic for status codes
Is it possible to do some custom logic based on status codes in the Puppeteer Crawler ? If yes, how ?
3 Replies
You can check the status code in the
response object from the PuppeteerCrawlingContext and do some logic based on that: https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#responseFor example if the status code is 403 it gets retried before I have the chance to check this response object
mere-teal•16mo ago
In the session pool options, you can specify what status codes should be treated as blocked requests. I think 403 is there by default, so passing an empty array will allow you full control here. Look this up - https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes