Hi vladdy VaclavRut Petr Pátek 2496
Hi @vladdy @Nardack @petrpatek.
If I need to do api(POST request) scraping using crawlee, how do I do it?
Thanks
If I need to do api(POST request) scraping using crawlee, how do I do it?
Thanks
method: "POST", payload: [STRING]{ proxyConfiguration: { useApifyProxy: false, proxyUrls: [...] }}const proxyConfiguration = await Actor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'FR',
});
const crawler = new HttpCrawler({
proxyConfiguration,
requestHandler: async ({ request, sendRequest, log, pushData}) => {
// based on the response of the post request I would like to make other post requests here to the same url but with diff headers and body.
log.debug(`Enqueueing pagination: ${request.url}`)
});
});
await crawler.run([
{
url: <URL>,
method: 'POST',
headers: headers,
payload: body,
},
]);// no opts
const proxyConfiguration = await Actor.createProxyConfiguration();
// opts
const proxyConfiguration = await Actor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'FR',
});
// and using
APIFY_PROXY_PASSWORD// with no options, does it autopicks a proxy? if yes, which group?
const proxyConfiguration = await Actor.createProxyConfiguration();
// with options I can understand the we are selecting a specific proxy
const proxyConfiguration = await Actor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'FR',
});
// or using
APIFY_PROXY_PASSWORD