No such file or directory storage/request_queues/default/JoxD7mAqj47ssmS.json

I'm trying to run a fairly simple scraper, but I keep getting this error. I want to scrape around 64,000 pages, but I get the no such file error every time. Setting waitForAllRequestsToBeAdded to true doesn't fix the issue.

This is how I'm setting up and running the crawler

const opts={
    navigationTimeoutSecs: 3,
    requestHandlerTimeoutSecs: 3,
    maxRequestRetries: 6,
    maxConcurrency: 20
};
const config=new Configuration({
    memoryMbytes: 8000
});
const crawler = new PlaywrightCrawler(opts, config);
crawler.router.addDefaultHandler(handlePage);
const requests = data.map(
    (d) =>
    new Request({
        url: d.url,
        userData: d
    })
);
await crawler.run(requests, {waitForAllRequestsToBeAdded: true});

const opts={
    navigationTimeoutSecs: 3,
    requestHandlerTimeoutSecs: 3,
    maxRequestRetries: 6,
    maxConcurrency: 20
};
const config=new Configuration({
    memoryMbytes: 8000
});
const crawler = new PlaywrightCrawler(opts, config);
crawler.router.addDefaultHandler(handlePage);
const requests = data.map(
    (d) =>
    new Request({
        url: d.url,
        userData: d
    })
);
await crawler.run(requests, {waitForAllRequestsToBeAdded: true});

Thanks for any help!

Apify & Crawlee•3y ago•

6 replies

colossal-harlequin

No such file or directory storage/request_queues/default/JoxD7mAqj47ssmS.json

const opts={
    navigationTimeoutSecs: 3,
    requestHandlerTimeoutSecs: 3,
    maxRequestRetries: 6,
    maxConcurrency: 20
};
const config=new Configuration({
    memoryMbytes: 8000
});
const crawler = new PlaywrightCrawler(opts, config);
crawler.router.addDefaultHandler(handlePage);
const requests = data.map(
    (d) =>
    new Request({
        url: d.url,
        userData: d
    })
);
await crawler.run(requests, {waitForAllRequestsToBeAdded: true});

const opts={
    navigationTimeoutSecs: 3,
    requestHandlerTimeoutSecs: 3,
    maxRequestRetries: 6,
    maxConcurrency: 20
};
const config=new Configuration({
    memoryMbytes: 8000
});
const crawler = new PlaywrightCrawler(opts, config);
crawler.router.addDefaultHandler(handlePage);
const requests = data.map(
    (d) =>
    new Request({
        url: d.url,
        userData: d
    })
);
await crawler.run(requests, {waitForAllRequestsToBeAdded: true});

Thanks for any help!

No such file or directory storage/request_queues/default/JoxD7mAqj47ssmS.json

Similar Threads

No such file or directory storage/request_queues/default/JoxD7mAqj47ssmS.json

Similar Threads

Similar Threads

Similar Threads