NeoNomade•2mo ago

re-enqueue request without throwing error

Is there any method of doing a request again without throwing an error but also respecting the maximum retries ?

3 Replies

Isaac Olads•2mo ago

Yes, there is a method you can go by, you can retry a request on failure while respecting a maximum retry limit

NeoNomadeOP•2mo ago

And what method would that be ?

Lukas Celnar•2mo ago

Hi @NeoNomade If you just want to suppress the error message it is easier to hack it like this:

const originalErrorLog = crawler.log.error.bind(crawler.log);
    crawler.log.error = (message: string, data?: Record<string, unknown> | null) => {
        if (message.includes(HIDE_MESSAGE_WATERMARK)) {
            return;
        }
        originalErrorLog(message, data);
    };

const originalErrorLog = crawler.log.error.bind(crawler.log);
    crawler.log.error = (message: string, data?: Record<string, unknown> | null) => {
        if (message.includes(HIDE_MESSAGE_WATERMARK)) {
            return;
        }
        originalErrorLog(message, data);
    };

Other option is to enqueue a new request and update its retryCount and uniqueKey

re-enqueue request without throwing error

Did you find this page helpful?