Downloading an image using puppeteer example

At a glance

The community member is looking to incorporate a Puppeteer example that downloads images as a page is visited into their Crawlee scraper. The example code is provided, and a community member suggests that the code is from a specific step in a tutorial. Another community member provides an updated version of the code that can be used as a preNavigationHooks function in Crawlee to download images. There is no explicitly marked answer.

Useful resources

MMichal

Hi, I found this simple example using Puppeteer, that downloads images as you visit the page and I'm wonderign how I can encorporate it into my crawlee scraper.

Plain Text

    this.page.on('response', async (response) => {
      const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

      console.log(matches);

      if (matches && (matches.length === 2)) {
        const extension = matches[1];
        const buffer = await response.buffer();

        fs.writeFileSync(`downloads/${this.request.userData}.${extension}`, buffer, 'base64');
        counter += 1;
      }
    });

2 comments

MMichal

It's from here, step 3 https://www.bannerbear.com/blog/how-to-download-images-from-a-website-using-puppeteer/

MMichal

Okay I figured it out:

Plain Text

      preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
          const { page, request } = crawlingContext;

          console.log('handlePageFunction');

          page.on('response', async (response) => {
            const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

            if (matches && (matches.length === 2)) {
              const extension = matches[1];
              const buffer = await response.buffer();

              fs.writeFileSync(
                `downloads/${request.userData.row.id}.${extension}`, 
                buffer,
                'base64'
              );
            }
          })
        }
      ]

Add a reply

Apify Discord Mirror

Downloading an image using puppeteer example