11 replies

Making API request with crawlee?

I need to make an api POST request to retrieve information in the body . I tried using the Basic crawler, since it uses Got-scrapping under the hood. I've had no success and I optedfor installing an axios package instead.

The code

 import { BasicCrawler} from 'crawlee'; 
// import { stored_data}  from './cookies.js';


const crawler = new BasicCrawler({
    async requestHandler ({ sendRequest , log }){
        const response = await sendRequest ({
            url: 'https://intouch.techdata.com/Intouch/ProductFE/api/ProductSearch/Search',
            method: 'POST',
            headers: { 
                hello:'hello',
                referer:'https://intouch.techdata.com/InTouch/MVC/ProductSearch/Search?tscv=4294949918',
            },
            json: {"TaxonomySubClassValue":4294950346,"DimensionValueIds":[4294950346],"SpecialistId":"1","SpecialistName":"BroadLine","IsWestminster":false,"SortBy":0,"ResultPreview":0,"RecordsPageSize":20,"RecordsOffset":2,"ShowTopSellersFromEndeca":true},
            responseType:'json' 
        });
        log.info(response.body);
    }
});

await crawler. Run()

 import { BasicCrawler} from 'crawlee'; 
// import { stored_data}  from './cookies.js';


const crawler = new BasicCrawler({
    async requestHandler ({ sendRequest , log }){
        const response = await sendRequest ({
            url: 'https://intouch.techdata.com/Intouch/ProductFE/api/ProductSearch/Search',
            method: 'POST',
            headers: { 
                hello:'hello',
                referer:'https://intouch.techdata.com/InTouch/MVC/ProductSearch/Search?tscv=4294949918',
            },
            json: {"TaxonomySubClassValue":4294950346,"DimensionValueIds":[4294950346],"SpecialistId":"1","SpecialistName":"BroadLine","IsWestminster":false,"SortBy":0,"ResultPreview":0,"RecordsPageSize":20,"RecordsOffset":2,"ShowTopSellersFromEndeca":true},
            responseType:'json' 
        });
        log.info(response.body);
    }
});

await crawler. Run()

Making API request with crawlee?

Similar Threads