Downloading JSON and YAML files while crawling with Playwright

slow-yellow · 2024-11-28T00:33:19.570Z

Hi there. Is it possible to detect the `Content-Type` header of responses and download JSON or YAML files? I'm using Playwright to crawl my sites and have some JSON and YAML content I would like to capture, as well.

H

Hall•11/28/24, 12:33 AM

Someone will reply to you shortly. In the meantime, this might help:

E

Exp•11/28/24, 12:54 AM

Yes of course

M

MEE6•11/28/24, 12:54 AM

@Exp just advanced to level 8! Thanks for your contributions!

S

slow-yellowOP•11/28/24, 1:40 AM

Thanks, @Exp. What's the best way of accomplishing that? I'm having trouble finding it in the API docs.

E

Exp•11/28/24, 2:25 AM

When you use the Playwright crawlee, you can capture all https request

E

Exp•11/28/24, 2:25 AM

Did you try it?

S

slow-yellowOP•11/28/24, 8:13 AM

Before the request handler fires, Playwright waits for the load event. When the crawler encounters JSON or YAML content types, the load event never fires, of course, and the request is aborted. I’m trying to figure out how I can intercept the crawler before that happens and either capture the content or queue it up for a different crawler.

A

azzouzana•11/28/24, 9:49 PM

@kevinswiber You can use https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#preNavigationHooks for this case. Something like this in js:

preNavigationHooks:[
    async (crawlingContext) => {
        crawlingContext.page.on('response', async function (response) {
                if (response.url().includes('......') {
                    const body = await response.json();
                   // check content type here and decide what to do here
                    const headers = response.headers();
                    const body = await response.json() // or text()
                }
            }
        )
    }
]

preNavigationHooks:[
    async (crawlingContext) => {
        crawlingContext.page.on('response', async function (response) {
                if (response.url().includes('......') {
                    const body = await response.json();
                   // check content type here and decide what to do here
                    const headers = response.headers();
                    const body = await response.json() // or text()
                }
            }
        )
    }
]

preNavigationHooks:[
    async (crawlingContext) => {
        crawlingContext.page.on('response', async function (response) {
                if (response.url().includes('......') {
                    const body = await response.json();
                   // check content type here and decide what to do here
                    const headers = response.headers();
                    const body = await response.json() // or text()
                }
            }
        )
    }
]

preNavigationHooks:[
    async (crawlingContext) => {
        crawlingContext.page.on('response', async function (response) {
                if (response.url().includes('......') {
                    const body = await response.json();
                   // check content type here and decide what to do here
                    const headers = response.headers();
                    const body = await response.json() // or text()
                }
            }
        )
    }
]

S

slow-yellowOP•12/1/24, 1:25 AM

@azzouzana Thank you! That was helpful. I feel the best solution would probably be creating a new crawler that extends the Playwright crawler. But in the meantime, I'm using this:

  preNavigationHooks: [
    async ({ request, page, log }) => {
      page.once("response", async (response) => {
        const contentType = response.headers()["content-type"];
        if (
          contentType?.includes("application/json") ||
          contentType?.includes("application/x-yaml") ||
          contentType?.includes("text/yaml") ||
          contentType?.includes("application/yaml")
        ) {
          const type = contentType.includes("json") ? "json" : "yaml";

          request.skipNavigation = true;
          request.noRetry = true;

          await new Promise(async (resolve, reject) => {
            page.once("download", async (download) => {
              const path = await download.path();

              const content = await readFile(path, "utf8");
              await dataset.pushData({
                url: download.url(),
                contentType,
                filename: download.suggestedFilename(),
                type,
                content,
              });
              log.info(`Downloaded ${type}: ${download.url()}`);
              resolve();
            });
            await page.waitForEvent("download");
          });
        }
      });
    },
  ],

  preNavigationHooks: [
    async ({ request, page, log }) => {
      page.once("response", async (response) => {
        const contentType = response.headers()["content-type"];
        if (
          contentType?.includes("application/json") ||
          contentType?.includes("application/x-yaml") ||
          contentType?.includes("text/yaml") ||
          contentType?.includes("application/yaml")
        ) {
          const type = contentType.includes("json") ? "json" : "yaml";

          request.skipNavigation = true;
          request.noRetry = true;

          await new Promise(async (resolve, reject) => {
            page.once("download", async (download) => {
              const path = await download.path();

              const content = await readFile(path, "utf8");
              await dataset.pushData({
                url: download.url(),
                contentType,
                filename: download.suggestedFilename(),
                type,
                content,
              });
              log.info(`Downloaded ${type}: ${download.url()}`);
              resolve();
            });
            await page.waitForEvent("download");
          });
        }
      });
    },
  ],

  preNavigationHooks: [
    async ({ request, page, log }) => {
      page.once("response", async (response) => {
        const contentType = response.headers()["content-type"];
        if (
          contentType?.includes("application/json") ||
          contentType?.includes("application/x-yaml") ||
          contentType?.includes("text/yaml") ||
          contentType?.includes("application/yaml")
        ) {
          const type = contentType.includes("json") ? "json" : "yaml";

          request.skipNavigation = true;
          request.noRetry = true;

          await new Promise(async (resolve, reject) => {
            page.once("download", async (download) => {
              const path = await download.path();

              const content = await readFile(path, "utf8");
              await dataset.pushData({
                url: download.url(),
                contentType,
                filename: download.suggestedFilename(),
                type,
                content,
              });
              log.info(`Downloaded ${type}: ${download.url()}`);
              resolve();
            });
            await page.waitForEvent("download");
          });
        }
      });
    },
  ],

  preNavigationHooks: [
    async ({ request, page, log }) => {
      page.once("response", async (response) => {
        const contentType = response.headers()["content-type"];
        if (
          contentType?.includes("application/json") ||
          contentType?.includes("application/x-yaml") ||
          contentType?.includes("text/yaml") ||
          contentType?.includes("application/yaml")
        ) {
          const type = contentType.includes("json") ? "json" : "yaml";

          request.skipNavigation = true;
          request.noRetry = true;

          await new Promise(async (resolve, reject) => {
            page.once("download", async (download) => {
              const path = await download.path();

              const content = await readFile(path, "utf8");
              await dataset.pushData({
                url: download.url(),
                contentType,
                filename: download.suggestedFilename(),
                type,
                content,
              });
              log.info(`Downloaded ${type}: ${download.url()}`);
              resolve();
            });
            await page.waitForEvent("download");
          });
        }
      });
    },
  ],

A

azzouzana•12/1/24, 1:36 AM

Glad it helped @kevinswiber I recently faced a similar case where some of the requests might return raw json, I tried a much simpler approach & it's working (You can inspect the response & check its headers)

S

slow-yellowOP•12/1/24, 3:50 AM

@azzouzana Is this using the Playwright crawler?

Kkevinswiber @azzouzana Is this using the Playwright crawler?

A

azzouzana•12/1/24, 12:15 PM

@kevinswiber you're right. It's using Cheerio's