fascinating-indigo•3y ago

Can I use modules inside the `evaluate`?

All the title says, specifically, can I use other modules like enum like object in evaluateAll function? For example:

const Foo = {
  FIZZ: "FIZZ",
  BUZZ: "BUZZ",
  FIZZ_BUZZ: "FIZZ_BUZZ",
} as const

await $anything.evaluateAll($divs =>
  $divs.map($div =>
    const text = $div.querySelector(".something").innerHTML
    if (text.includes("fizz")) {
      return Foo.FIZZ // ERROR Because "Foo" is not in the browser JavaScript
    }
  )
)

const Foo = {
  FIZZ: "FIZZ",
  BUZZ: "BUZZ",
  FIZZ_BUZZ: "FIZZ_BUZZ",
} as const

await $anything.evaluateAll($divs =>
  $divs.map($div =>
    const text = $div.querySelector(".something").innerHTML
    if (text.includes("fizz")) {
      return Foo.FIZZ // ERROR Because "Foo" is not in the browser JavaScript
    }
  )
)

11 Replies

fascinating-indigoOP•3y ago

Oh, maybe should I ask this in playwright community

optimistic-gold•3y ago

I don't quite understand your question. But below is an example that works:

import {
  PlaywrightCrawler,
  log,
} from 'crawlee';

const crawler = new PlaywrightCrawler({
  headless: false,

  async requestHandler({ request, page, log }) {
    const title = await page.title();
    log.info(`Title of ${request.loadedUrl} is '${title}'`);

    const result = await page.evaluate(async () => {
      // Object.freeze() method freezes an object meaning new properties cannot be added,
      // existing properties, and the object's prototype cannot be re-assigned.
      const Foo = Object.freeze({
        FIZZ: "FIZZ",
        BUZZ: "BUZZ",
        FIZZ_BUZZ: "FIZZ_BUZZ",
      });

      const text =  document.querySelector("h1").innerHTML;
      if (text.includes("Example Domain")) {
        return(Foo.FIZZ);
      }
    });
    log.info(`requestHandler() Result: ${result}`);
  },
});

await crawler.run([{ url: 'https://example.com/' }]);

import {
  PlaywrightCrawler,
  log,
} from 'crawlee';

const crawler = new PlaywrightCrawler({
  headless: false,

  async requestHandler({ request, page, log }) {
    const title = await page.title();
    log.info(`Title of ${request.loadedUrl} is '${title}'`);

    const result = await page.evaluate(async () => {
      // Object.freeze() method freezes an object meaning new properties cannot be added,
      // existing properties, and the object's prototype cannot be re-assigned.
      const Foo = Object.freeze({
        FIZZ: "FIZZ",
        BUZZ: "BUZZ",
        FIZZ_BUZZ: "FIZZ_BUZZ",
      });

      const text =  document.querySelector("h1").innerHTML;
      if (text.includes("Example Domain")) {
        return(Foo.FIZZ);
      }
    });
    log.info(`requestHandler() Result: ${result}`);
  },
});

await crawler.run([{ url: 'https://example.com/' }]);

Pepa J•3y ago

I see an example in Playwright docs with passing out-scoped value as another parameter https://playwright.dev/docs/api/class-locator#locator-evaluate-all

const moreThanTen = await locator.evaluateAll((divs, min) => divs.length > min, 10);

const moreThanTen = await locator.evaluateAll((divs, min) => divs.length > min, 10);

at this example it is min = 10

optimistic-gold•3y ago

And you can't import modules inside the page, which does not know anything about where to import that file. See here for more information: https://playwright.dev/docs/evaluating

Evaluating JavaScript | Playwright

Playwright scripts run in your Playwright environment. Your page scripts run in the browser page environment. Those environments don't intersect, they are running in different virtual machines in different processes and even potentially on different computers.

optimistic-gold•3y ago

But Playwright's Context provides a way to inject new function in the browser instance: browserContext.exposeFunction(name, callback) https://playwright.dev/docs/api/class-browsercontext#browser-context-expose-function

BrowserContext | Playwright

* extends: [EventEmitter]

fascinating-indigoOP•3y ago

Maybe this would be help, thanks, gotta give a try. I need to define Foo enum outside the module. I know about it, well so I wrote the comment in the text. Thank you tho Thank you, this could be a solution.

MEE6•3y ago

@rikusen0335 just advanced to level 2! Thanks for your contributions! 🎉

Lukas Krivka•3y ago

@rikusen0335 Everything going inside evaluate is going through JSON.stringify/parse so you can only pass such values that survive this serialization

fascinating-indigoOP•3y ago

Oh, okay I got it. So the enum maybe cannot be used

Pepa J•3y ago

I believe Enums should be fine https://www.typescriptlang.org/play?#code/KYOwrgtgBAsgngUXNA3gKCpqA1AggGQFUEBGKAXigCI8jSqAaDLW4gJgutYTarQF80aAMYB7EAGdRAG2AA6aaIDmACgBSAZQDyAOTkSALgCcAliCUmAZnBXwkkAJQOgA since it is possible to serialize them.

TS Playground - An online editor for exploring TypeScript and JavaS...

The Playground lets you write TypeScript or JavaScript online in a safe and sharable way.

fascinating-indigoOP•3y ago

Oh thank you for confirming it! I really appreciate it:blobdance:

Can I use modules inside the `evaluate`?

Did you find this page helpful?