Crawlee doesn't process newly enqueued links via enqueueLinks

Hi folks, I'm trying to build a crawler that retrieves a body (Buffer), and later enqueues the next "page" to be crawled, if it exists (has_next === true ). The problem is that

?page=1

?page=1

gets processed but the enqueued page (via

enqueueLinks

enqueueLinks

) doesn't; Crawlee states that it has processed all links (1 of 1).

I have confirmed that

has_next

has_next

is indeed

true

true

and that

enqueueLinks

enqueueLinks

gets called.

Am I missing something obvious?

await crawler.run([
    { url: 'http://quotes.toscrape.com/api/quotes?page=1', label: 'quotes' },
])

// ...

routerRef.addHandler('quotes', async (context) => {

        const { request, sendRequest, enqueueLinks, response, body, log } = context

        log.info(request.url)
        if (!(body instanceof Buffer)) {
            log.error(`Expected a Buffer instance.`)
            return
        }

        const json: QuotesResponse = JSON.parse(body.toString())

        if ((json.quotes?.length ?? 0) > 0) {
            await datasetRef.pushData(json)
        }

        if (json.has_next) {
            const urls = [`http://quotes.toscrape.com/api/quotes?page=${json.page + 1}`]
            await enqueueLinks({
                urls,
                label: 'quotes',
                strategy: EnqueueStrategy.All,
            })
        } else {
            log.warning('No next was found')
        }

    })

await crawler.run([
    { url: 'http://quotes.toscrape.com/api/quotes?page=1', label: 'quotes' },
])

// ...

routerRef.addHandler('quotes', async (context) => {

        const { request, sendRequest, enqueueLinks, response, body, log } = context

        log.info(request.url)
        if (!(body instanceof Buffer)) {
            log.error(`Expected a Buffer instance.`)
            return
        }

        const json: QuotesResponse = JSON.parse(body.toString())

        if ((json.quotes?.length ?? 0) > 0) {
            await datasetRef.pushData(json)
        }

        if (json.has_next) {
            const urls = [`http://quotes.toscrape.com/api/quotes?page=${json.page + 1}`]
            await enqueueLinks({
                urls,
                label: 'quotes',
                strategy: EnqueueStrategy.All,
            })
        } else {
            log.warning('No next was found')
        }

    })

Apify & Crawlee•3y ago•

4 replies

specific-silver

Crawlee doesn't process newly enqueued links via enqueueLinks

Hi folks, I'm trying to build a crawler that retrieves a body (Buffer), and later enqueues the next "page" to be crawled, if it exists (has_next === true ). The problem is that

?page=1

?page=1

gets processed but the enqueued page (via

enqueueLinks

enqueueLinks

) doesn't; Crawlee states that it has processed all links (1 of 1).

I have confirmed that

has_next

has_next

is indeed

true

true

and that

enqueueLinks

enqueueLinks

gets called.

Am I missing something obvious?

await crawler.run([
    { url: 'http://quotes.toscrape.com/api/quotes?page=1', label: 'quotes' },
])

// ...

routerRef.addHandler('quotes', async (context) => {

        const { request, sendRequest, enqueueLinks, response, body, log } = context

        log.info(request.url)
        if (!(body instanceof Buffer)) {
            log.error(`Expected a Buffer instance.`)
            return
        }

        const json: QuotesResponse = JSON.parse(body.toString())

        if ((json.quotes?.length ?? 0) > 0) {
            await datasetRef.pushData(json)
        }

        if (json.has_next) {
            const urls = [`http://quotes.toscrape.com/api/quotes?page=${json.page + 1}`]
            await enqueueLinks({
                urls,
                label: 'quotes',
                strategy: EnqueueStrategy.All,
            })
        } else {
            log.warning('No next was found')
        }

    })

await crawler.run([
    { url: 'http://quotes.toscrape.com/api/quotes?page=1', label: 'quotes' },
])

// ...

routerRef.addHandler('quotes', async (context) => {

        const { request, sendRequest, enqueueLinks, response, body, log } = context

        log.info(request.url)
        if (!(body instanceof Buffer)) {
            log.error(`Expected a Buffer instance.`)
            return
        }

        const json: QuotesResponse = JSON.parse(body.toString())

        if ((json.quotes?.length ?? 0) > 0) {
            await datasetRef.pushData(json)
        }

        if (json.has_next) {
            const urls = [`http://quotes.toscrape.com/api/quotes?page=${json.page + 1}`]
            await enqueueLinks({
                urls,
                label: 'quotes',
                strategy: EnqueueStrategy.All,
            })
        } else {
            log.warning('No next was found')
        }

    })

Crawlee doesn't process newly enqueued links via enqueueLinks

Similar Threads

Crawlee doesn't process newly enqueued links via enqueueLinks

Similar Threads

Similar Threads

Similar Threads