NeoNomade•2y ago

Deprecation in Puppeteer

Hello, Puppeteer deprecated the page.waitForTimeout function. This resulted in making the infiniteScroll function from utils to be broken. Is there any fix in progress ?

12 Replies

Oleg V.•2y ago

Hey. Didn't hear any complaints about it recently. Can You please provide some repro / link to your run with mentioned issues. Will check / pass it to our team.

NeoNomadeOP•2y ago

I’m running locally . If you import utils. then await util.puppeteerUtils.infiniteScroll . It will throw error that await page.waitForTimeout doesn’t exist

Oleg V.•2y ago

at least copy-paste error trace then please

NeoNomadeOP•2y ago

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. TypeError: page.waitForTimeout is not a function
    at Object.infiniteScroll (/project/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:449:20)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file://project/src/routes.js:78:5
    at async wrap (/project/node_modules/@apify/timeout/cjs/index.cjs:54:21) {"id":"lO5wjYaCDed0lnp","url":"some_website","retryCount":1}

WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. TypeError: page.waitForTimeout is not a function
    at Object.infiniteScroll (/project/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:449:20)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file://project/src/routes.js:78:5
    at async wrap (/project/node_modules/@apify/timeout/cjs/index.cjs:54:21) {"id":"lO5wjYaCDed0lnp","url":"some_website","retryCount":1}

removed path and website.

package.json
❯ cat package.json 
{
    "name": "item_detail",
    "version": "0.0.1",
    "type": "module",
    "description": "This is an example of a Crawlee project.",
    "dependencies": {
        "crawlee": "^3.0.0",
        "puppeteer": "*",
        "utility-scripts": "file:_scripts"
    },
    "scripts": {
        "start": "node src/main.js",
        "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
    },
    "author": "It's not you it's me",
    "license": "ISC"
}

package.json
❯ cat package.json 
{
    "name": "item_detail",
    "version": "0.0.1",
    "type": "module",
    "description": "This is an example of a Crawlee project.",
    "dependencies": {
        "crawlee": "^3.0.0",
        "puppeteer": "*",
        "utility-scripts": "file:_scripts"
    },
    "scripts": {
        "start": "node src/main.js",
        "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
    },
    "author": "It's not you it's me",
    "license": "ISC"
}

the utility_scripts, are just some database communications and some axios scripts, they don't interfere with puppeteer or crawlee. the call from line 78:

    await puppeteerUtils.infiniteScroll(page, {
        timeoutSecs: 0,
        scrollDownAndUp: true,
        waitForSecs: 10,
        buttonSelector: 'div[id="store-wrapper"]  div[role="none"] button',
    });

    await puppeteerUtils.infiniteScroll(page, {
        timeoutSecs: 0,
        scrollDownAndUp: true,
        waitForSecs: 10,
        buttonSelector: 'div[id="store-wrapper"]  div[role="none"] button',
    });

are there any other details that you need ?

other-emerald•2y ago

This is already fixed in beta: https://github.com/apify/crawlee/commit/52d7219acdc19b34a727e5d26f7f9288d27ca57f (not yet released as latest)

GitHub

fix(puppeteer): replace page.waitForTimeout() with sleep() · ap...

Closes #2335

NeoNomadeOP•2y ago

@Andrey Bykov how can I somehow force npm to install this beta ?

other-emerald•2y ago

just npm install crawlee@next

NeoNomadeOP•2y ago

Thanks !

other-emerald•2y ago

just a heads-up, it was released as latest

NeoNomadeOP•2y ago

The fix is now in latest ?

other-emerald•2y ago

yup

other-emerald•2y ago

https://github.com/apify/crawlee/releases/tag/v3.8.0

GitHub

Release v3.8.0 · apify/crawlee

3.8.0 (2024-02-21) Bug Fixes createRequests works correctly with exclude (and nothing else) (#2321) (048db09) declare missing dependencies on csv-stringify and fs-extra (#2326) (718959d), closes /...

Deprecation in Puppeteer

Did you find this page helpful?