Apify Discord Mirror

Updated 5 months ago

custom logic for status codes

At a glance

The post asks if it is possible to do custom logic based on status codes in the Puppeteer Crawler, and how to do it. The community members provide the following information:

You can check the status code in the response object from the PuppeteerCrawlingContext and do some logic based on that. For example, if the status code is 403, it gets retried before you have a chance to check the response object.

In the session pool options, you can specify what status codes should be treated as blocked requests. 403 is there by default, so passing an empty array will allow you full control here.

Useful resources
Is it possible to do some custom logic based on status codes in the Puppeteer Crawler ? If yes, how ?
R
N
m
3 comments
You can check the status code in the response object from the PuppeteerCrawlingContext and do some logic based on that: https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#response
For example if the status code is 403 it gets retried before I have the chance to check this response object
In the session pool options, you can specify what status codes should be treated as blocked requests. I think 403 is there by default, so passing an empty array will allow you full control here. Look this up - https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes
Add a reply
Sign up and join the conversation on Discord