wise-white•14mo ago
How do I add query parameter to globbed links in enqueueLinks?
How do I add a query parameter to globbed links in enqueueLinks? For example I want
http://www.foo.com/bar -> http://www.foo.com/bar?qux=1. I tried requestTransform but that seems to transform all requests, even ones that don't match the glob. Do I need to re-check the URL in the requestTransform function and conditionally modify the url? Is there no other way to do this?1 Reply
Hi @drasticstuff
You could try something like this:
await enqueueLinks({
requestTransform: (requestOptions) => {
const url = new URL(requestOptions.url);
if (url.href.startsWith('http://www.foo.com/bar')) {
url.searchParams.set('qux', '1');
requestOptions.url = url.href;
}
return requestOptions;
},
});
Or you can scrape the urls and enque them them manually with await crawler.addRequests([{ url: ${url}?qux=1 }])