Is it possible to run a crawl within a crawl?

I have a crawler I have setup (x) that collects links from a page. I want to run another crawler(y) within x crawl to collect more data and return it with the rest of x crawl data. Is it possible to do this?

(I know you can add requests to the queue from a crawl but I want to keep the data together)

example data structure:

const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]

const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]

Apify & Crawlee•3y ago•

4 replies

spotty-amber

Is it possible to run a crawl within a crawl?

const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]

const data = [{
    // from "x" crawl
    title: 'a page',
    links: [
        {
            // from "y" crawl
            title: 'a page',
            link: 'link'
        }
    ]
}]

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

15,108 Members

Join

Is it possible to run a crawl within a crawl?

Is it possible to run a crawl within a crawl?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads