Module not found in NextJs projects

At a glance

The community member is trying to run a Playwright crawler with Crawlee, but is encountering an error saying the Puppeteer module cannot be found. They have installed Crawlee and Playwright, but not Puppeteer. The community members discuss that for Next.js projects, Crawlee and Playwright need to be added to the Webpack externals array. However, there are still issues with running Playwright in Edge functions and Vercel serverless functions.

Useful resources

ttim

I am trying to run a playwright crawler with Crawlee and for some reason I'm getting a error saying the puppeteer module cannot be found. I've installed npm install crawlee playwright but not puppeteer

Do i need puppeteer also to run playwright? i dont think so...

Plain Text

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/@crawlee/puppeteer/index.js
./node_modules/crawlee/index.js
./node_modules/crawlee/index.mjs
./app/actions/crawlee.ts
./app/api/local/crawlee/route.ts
 ⨯ ./node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:190:21
Module not found: Can't resolve 'puppeteer/package.json'

Attachment

5 comments

ttim

SOLUTION FOUND:
For nextjs projects, add 'crawlee' and 'playwright' to the webpack externals array like this

Attachment

ttim

Plain Text

webpack: (config, { isServer }) => {
    if (isServer) {
      config.externals.push(
        'crawlee',
        'puppeteer',
        'playwright',
        // ... any other server-side only packages
      );
    }
    return config;
  },

ttim

Module not found in NextJs projects

ggablabelle

Hello were you able to make this work in an Edge function? I get Error [ReferenceError]: crawlee is not defined when in edge functions. OK in normal serverless API routes.

ggablabelle

Actually when deployed it doesn't work in normal serverless functions. I get [cause]: Error: Cannot find module 'playwright'.

were you able to make playwright run in vercel serverless functions at all?

Add a reply

Apify Discord Mirror

Module not found in NextJs projects