automatic-azure•4y ago

http response status

ist there any way to handle a response status explicitly? (puppeteer/cheerio) there seems to be a response parameter in the requesthandler but I don't know how to use that. so e.g looking to handle a 400 response code...

10 Replies

blank-aquamarine•4y ago

const crawler = new CheerioCrawler({
    requestHandler: ({ response }) => {
        console.log(response.statusCode);
    }
})

const crawler = new CheerioCrawler({
    requestHandler: ({ response }) => {
        console.log(response.statusCode);
    }
})

automatic-azureOP•4y ago

Thanks @thek1tten

    async requestHandler({
      request, page, log, response,
    }) {
      console.log(response.statusCode);

Cannot read properties of undefined (reading 'statusCode') I'm using a PuppeteerCrawler, does this make a difference? Obviously ist does. Seems like response object doesn't have a statusCode in PuppeteerCrawler, works in Cheerio

automatic-azureOP•4y ago

https://github.com/apify/crawlee/issues/1557

GitHub

response status missing in puppeteer crawler (works in cheerio) · I...

Which package is this bug report for? If unsure which one to select, leave blank @crawlee/puppeteer (PuppeteerCrawler) Issue description The response.statusCode is missing in Puppeteer crawler This...

blank-aquamarine•4y ago

PuppeteerCrawler returns a Puppeteer HTTPResponse object. https://pptr.dev/next/api/puppeteer.httpresponse/ Grab the status from it with the .status() function.

import { PuppeteerCrawler } from 'crawlee';

const crawler = new PuppeteerCrawler({
    requestHandler: ({ response }) => {
        console.log(response.status());
    }
})

import { PuppeteerCrawler } from 'crawlee';

const crawler = new PuppeteerCrawler({
    requestHandler: ({ response }) => {
        console.log(response.status());
    }
})

HTTPResponse class | Puppeteer

The HTTPResponse class represents responses which are received by the Page class.

automatic-azureOP•4y ago

Thanks Matt! this helps 🙂

Lukas Krivka•4y ago

You can also adjust the blocked status code of sessionPool https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes

SessionPoolOptions | API | Crawlee

automatic-azureOP•4y ago

@Lukas Krivka if I block it I can't handle it in my code anymore, right?

blank-aquamarine•4y ago

@bs Correct. For example, if you add the 200 status code to the blocked list, the request will immediately retry and the requestHandler for that request will not be executed.

automatic-azureOP•4y ago

Thanks @thek1tten that's what i thought. I'd rather handle the different status codes by myself 🙂

blank-aquamarine•4y ago

So you can make the list empty 😄

http response status

Did you find this page helpful?