Could not find file at storage/key_value_stores/default/SDK_SESSION_POOL_STATE.json

Hi there!

I'm crawling some pages from different countries using proxy configurations.

Function running crawling:

async crawl(proxyUrl: string, sites: string[]) {
    const config = new Configuration({
      disableBrowserSandbox: true,
    });

    const proxyConfiguration = new ProxyConfiguration({
      proxyUrls: [proxyUrl],
    });

    const urlsHash: Record<string, string[]> = {};

    const crawler = new PuppeteerCrawler(
      {
        proxyConfiguration,
        maxRequestRetries: 0,
        requestHandler: async ({ page }) => {
          const pageUrl = page.url();

          page.on('request', (request) => {
            ...
          });

          await this.autoScroll(page);

          await page.waitForNetworkIdle({
            idleTime: 1000,
          });
        },
      },
      config,
    );

    await crawler.run(sites);

    return urlsHash;
  }

async crawl(proxyUrl: string, sites: string[]) {
    const config = new Configuration({
      disableBrowserSandbox: true,
    });

    const proxyConfiguration = new ProxyConfiguration({
      proxyUrls: [proxyUrl],
    });

    const urlsHash: Record<string, string[]> = {};

    const crawler = new PuppeteerCrawler(
      {
        proxyConfiguration,
        maxRequestRetries: 0,
        requestHandler: async ({ page }) => {
          const pageUrl = page.url();

          page.on('request', (request) => {
            ...
          });

          await this.autoScroll(page);

          await page.waitForNetworkIdle({
            idleTime: 1000,
          });
        },
      },
      config,
    );

    await crawler.run(sites);

    return urlsHash;
  }

After the first run finished(and a promise is resolved), I call this function again with another params, but get the following error:
Could not find file at /usr/src/app/storage/key_value_stores/default/SDK_SESSION_POOL_STATE.jsonCould not find file at /usr/src/app/storage/key_value_stores/default/SDK_SESSION_POOL_STATE.json

Could someone help what is wrong?

I suppose that a session after the first run was cleared and the file from the error message was deleted, but it seems like it expects to have this file.

Apify & Crawlee•3y ago•

14 replies

slow-yellow

Could not find file at storage/key_value_stores/default/SDK_SESSION_POOL_STATE.json

Hi there!

I'm crawling some pages from different countries using proxy configurations.

Function running crawling:

async crawl(proxyUrl: string, sites: string[]) {
    const config = new Configuration({
      disableBrowserSandbox: true,
    });

    const proxyConfiguration = new ProxyConfiguration({
      proxyUrls: [proxyUrl],
    });

    const urlsHash: Record<string, string[]> = {};

    const crawler = new PuppeteerCrawler(
      {
        proxyConfiguration,
        maxRequestRetries: 0,
        requestHandler: async ({ page }) => {
          const pageUrl = page.url();

          page.on('request', (request) => {
            ...
          });

          await this.autoScroll(page);

          await page.waitForNetworkIdle({
            idleTime: 1000,
          });
        },
      },
      config,
    );

    await crawler.run(sites);

    return urlsHash;
  }

async crawl(proxyUrl: string, sites: string[]) {
    const config = new Configuration({
      disableBrowserSandbox: true,
    });

    const proxyConfiguration = new ProxyConfiguration({
      proxyUrls: [proxyUrl],
    });

    const urlsHash: Record<string, string[]> = {};

    const crawler = new PuppeteerCrawler(
      {
        proxyConfiguration,
        maxRequestRetries: 0,
        requestHandler: async ({ page }) => {
          const pageUrl = page.url();

          page.on('request', (request) => {
            ...
          });

          await this.autoScroll(page);

          await page.waitForNetworkIdle({
            idleTime: 1000,
          });
        },
      },
      config,
    );

    await crawler.run(sites);

    return urlsHash;
  }

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

Could not find file at storage/key_value_stores/default/SDK_SESSION_POOL_STATE.json

Could not find file at storage/key_value_stores/default/SDK_SESSION_POOL_STATE.json

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads