Scrape redirect links gracefully?

Hey folks,

I have a site which gives out internal links which then redirects to the actual page and I want to scrape the redirected link, I tried using the

sendRequest

sendRequest

api but the response's url only had the internal url and not the actual link, so now I am doing it via playwright like this

        const context = page.context();
        const newPagePromise = context.waitForEvent('page');
        await applyNowBtn.click();
        const newPage = await newPagePromise;
        await newPage.waitForTimeout(3000)
        log.info(`url is ${newPage.url()}`);
        newPage.close();

        const context = page.context();
        const newPagePromise = context.waitForEvent('page');
        await applyNowBtn.click();
        const newPage = await newPagePromise;
        await newPage.waitForTimeout(3000)
        log.info(`url is ${newPage.url()}`);
        newPage.close();

issue with this approach is (aside from having to use a timeout), crawlee logs these

cant set cookie

cant set cookie

errors which adds a lot of unnecessary noise to my logs, I just wanted to know if there was a better approach to handle this?

Apify & Crawlee•3y ago•

4 replies

verbal-lime

Scrape redirect links gracefully?

Hey folks,

I have a site which gives out internal links which then redirects to the actual page and I want to scrape the redirected link, I tried using the

sendRequest

sendRequest

api but the response's url only had the internal url and not the actual link, so now I am doing it via playwright like this

        const context = page.context();
        const newPagePromise = context.waitForEvent('page');
        await applyNowBtn.click();
        const newPage = await newPagePromise;
        await newPage.waitForTimeout(3000)
        log.info(`url is ${newPage.url()}`);
        newPage.close();

        const context = page.context();
        const newPagePromise = context.waitForEvent('page');
        await applyNowBtn.click();
        const newPage = await newPagePromise;
        await newPage.waitForTimeout(3000)
        log.info(`url is ${newPage.url()}`);
        newPage.close();

issue with this approach is (aside from having to use a timeout), crawlee logs these

cant set cookie

cant set cookie

errors which adds a lot of unnecessary noise to my logs, I just wanted to know if there was a better approach to handle this?

Scrape redirect links gracefully?

Similar Threads

Scrape redirect links gracefully?

Similar Threads

Similar Threads

Similar Threads