5 replies

Is there a way to enqueue a request without actually making an HTTP request?

🌐HttpCrawler

What I’m trying to do is leverage Crawlee’s request queue + uniqueKey de-duplication, but I don’t want to hit a real URL.

Conceptually, I just want to “switch routes” (change the label) and trigger another operation/state in the crawler, while still using the queue’s uniqueness guarantees.

Is this possible, or is a real URL always required when adding a request?

Currently getting

 Reclaiming failed request back to the list or queue. Expected `RequestOptions.url` to be of type `string` but received type `undefined`

 Reclaiming failed request back to the list or queue. Expected `RequestOptions.url` to be of type `string` but received type `undefined`

I want something like that

import { v4 as uuidv4 } from 'uuid';

const itemRequests: Partial<Source>[] = dataResults.map((result) => {
    const itemKey = result.item.key; // generic key
    return {
        uniqueKey: request.userData.saveOnlyUniqueItems ? itemKey : uuidv4(),
        label: labels.SPECIFIC_ITEM, // generic label
        userData: {
            ...request.userData,
            itemData: result.item, // pass the data generically
        },
    };
});

import { v4 as uuidv4 } from 'uuid';

const itemRequests: Partial<Source>[] = dataResults.map((result) => {
    const itemKey = result.item.key; // generic key
    return {
        uniqueKey: request.userData.saveOnlyUniqueItems ? itemKey : uuidv4(),
        label: labels.SPECIFIC_ITEM, // generic label
        userData: {
            ...request.userData,
            itemData: result.item, // pass the data generically
        },
    };
});

Apify & Crawlee•5mo ago•

5 replies

fierDeToiMonGrand

Is there a way to enqueue a request without actually making an HTTP request?

🌐HttpCrawler

 Reclaiming failed request back to the list or queue. Expected `RequestOptions.url` to be of type `string` but received type `undefined`

 Reclaiming failed request back to the list or queue. Expected `RequestOptions.url` to be of type `string` but received type `undefined`

I want something like that

import { v4 as uuidv4 } from 'uuid';

const itemRequests: Partial<Source>[] = dataResults.map((result) => {
    const itemKey = result.item.key; // generic key
    return {
        uniqueKey: request.userData.saveOnlyUniqueItems ? itemKey : uuidv4(),
        label: labels.SPECIFIC_ITEM, // generic label
        userData: {
            ...request.userData,
            itemData: result.item, // pass the data generically
        },
    };
});

import { v4 as uuidv4 } from 'uuid';

const itemRequests: Partial<Source>[] = dataResults.map((result) => {
    const itemKey = result.item.key; // generic key
    return {
        uniqueKey: request.userData.saveOnlyUniqueItems ? itemKey : uuidv4(),
        label: labels.SPECIFIC_ITEM, // generic label
        userData: {
            ...request.userData,
            itemData: result.item, // pass the data generically
        },
    };
});

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,851 Members

Join

Is there a way to enqueue a request without actually making an HTTP request?

Is there a way to enqueue a request without actually making an HTTP request?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads