Crawler for SPAs (Single Page Application)

Hi all!

My target is to scrap a website composed of SPAs (Single Page Application) and it looks like existing browser crawlers (i.e. PlaywrightCrawler and PuppeteerCrawler) are not a good fit as each request is processed in a new page, which is a waste of resources.

What I need is to open one browser page and execute multiple XHR / fetch requests to their unofficial API, until I get blocked and need to re-open a new browser page to continue until all requests have been processed.

Note that need a browser to pass fingerprint checks and use the website's internal library to digitally sign each request to their unofficial API.

I'm thinking to solve my need by writing a

SinglePageBrowserCrawler

SinglePageBrowserCrawler

that extends BasicCrawler and works similarly to BrowserCrawler but manage browser pages differently.

Is it a good idea? Is there a way to do this in a better way?

Thanks in advance for your feedback!

PlaywrightCrawler | API | Crawlee

Provides a simple framework for parallel crawling of web pages
using headless Chromium, Firefox and Webkit browsers with Playwright.
The URLs to crawl are fed either from a static list of URLs
or from a dynamic queue of URLs enabling recursive crawling of websites.

Since

Playwright

Playwright

uses headless brow...

MDN Web Docs

XMLHttpRequest (XHR) - MDN Web Docs Glossary: Definitions of Web-re...

XMLHttpRequest (XHR) is a JavaScript API to create HTTP requests. Its methods provide the ability to send network requests between the browser and a server.

MDN Web Docs

fetch() global function - Web APIs | MDN

The global fetch() method starts the process of fetching a resource from the network, returning a promise that is fulfilled once the response is available.

BasicCrawler | API | Crawlee

Provides a simple framework for parallel crawling of web pages.
The URLs to crawl are fed either from a static list of URLs
or from a dynamic queue of URLs enabling recursive crawling of websites.

BasicCrawler

BasicCrawler

is a low-level tool that requires the user to implement the page
download and data extraction functionality themselves.
If we want a c...

sad-indigo•1/8/24, 4:36 PM

Overall, to my experience, it's better to try to reuse existing solutions as maximum as possible, before implementing something by hand.
I'm not very proficient with Crawlee, but would like to draw attention to couple of things that could help you. There is function (from Got Scrapping), which could be used for inner requests (while Playwright loads the main request page). It's available in the CrawlingContext same as ,

log

log

, , etc.
Also, you could add these API requests to the Request Queue and in your handler retrieve the webpage's body and do something like (not precise syntax, but the idea)

sad-indigo•1/8/24, 5:01 PM

Also just find out about the parameter, so you could speed up the API crawling using a RequestQueue without openning the pages in a browser (docs)

unique-blueOP•1/9/24, 2:20 AM

Thanks @Mike , I agree with you, I also don't like to waste time to re-invent the wheel. That's why I share my problem here

I tried to use and already, but the PlaywrightCrawler or PuppeteerCrawler open and close a page for each request.

This is a problem for me, because each request I make to the website's internal API needs to be digitally signed via an algorithm contained in the website page. I tried to use but I was always blocked, even when I passed the headers and cookies, I suspect that the digital signature expires quickly (I might also have missed something, the

fetch

fetch

function is overridden by the website to add this digital signature).

In the meantime I made a class that extends the BrowserCrawler and override (where the page is opened) and (where the page is closed). I'm very grateful for Apify to open source and document Crawlee, as I was able to come up with this solution relatively quickly, but again there might be a better way to do that.

I also tried another solution: I gave only one request to the crawler, to open the homepage of the website I want to scrap, and saved all my API calls in the userData:

The problem with this solution is that I have to manage the actual requests all by myself, so I can't leverage Crawlee features such as parallelization and auto-retry.

BrowserCrawler | API | Crawlee

Provides a simple framework for parallel crawling of web pages
using headless browsers with Puppeteer
and Playwright.
The URLs to crawl are fed either from a static list of URLs
or from a dynamic queue of URLs enabling recursive crawling of websites.

Since `Bro...

RequestOptions | API | Crawlee

Specifies required and optional fields for constructing a {@apilink Request}.

memo23•1/9/24, 9:58 PM

@Marc Plouhinec interesting topic, in my case I created two crawlers one with Puppeteer for login and saving necessary details and those details I inject into second Basic crawler which finishes the job, seems fine to me

unique-blueOP•1/10/24, 1:31 AM

@memo23 It's a smart strategy, unfortunately in my case I need to keep the puppeteer page open as I need to constantly invoke functions inside this page to digitally sign my forged HTTP requests to their API.

memo23•1/10/24, 4:39 AM

If I don’t use crawlee, what I usually do with puppeteer is in page.evaluate I fire jusj ajax request amd when it fails I just fix with puppeteer and continue

Lukas Krivka•1/12/24, 7:09 PM

Generally, we solve SPAs with CheerioCrawler, the data is either in initial HTML in scripts and/or in XHRs that you need to compose. Sometimes that might be tricky and requires some experience. https://docs.apify.com/academy/api-scraping

If that would be too difficult, you can always fallback to browsers and clicking around.

API scraping | Academy | Apify Documentation

Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.

unique-blueOP•1/13/24, 4:13 AM

@Lukas Krivka A CheerioCrawler would be the best, but in my case the website's backend APIs requires signed HTTP requests. So I would have to replicate their signature algorithm in order to send my HTTP requests from NodeJS.

I tried this solution initially, by extracting the website's Webpack bundle responsible for signing requests, and then running it in a NodeJS context. But I eventually abandoned this solution as the signature was rejected (I must have missed something).

I finally implemented a

SinglePageBrowserCrawler

SinglePageBrowserCrawler

that keeps pages open across requests (as long as no blocking is detected) and run to sign and execute fetch/XHR requests . It's an acceptable compromise for now: although it's more resource intensive than a CheerioCrawler, it's much faster than clicking around, and also more flexible as I can tweak the API parameters (e.g. more results per page).

If I have time in the future I will try again to convert my actor into a CheerioCrawler.

Digital signature

A digital signature is a mathematical scheme for verifying the authenticity of digital messages or documents. A valid digital signature on a message gives a recipient confidence that the message came from a sender known to the recipient.Digital signatures are a standard element of most cryptographic protocol suites, and are commonly used for sof...

Lukas Krivka•1/13/24, 8:14 AM

Yeah, we also sometimes give up on reverse engineering if it takes few hours with not much progress. Would you mind sharing core of the

SinglePageBrowserCrawler

SinglePageBrowserCrawler

code?, it seems like an elegant solution.

unique-blueOP•1/13/24, 8:36 AM

Sure! not sure it's elegant as I had to copy/paste code from BrowserCrawler and PuppeteerCrawler, but here you go!

unique-blueOP•1/13/24, 8:42 AM

What would be great is to have multiple "modes" in the existing in Crawlee:

open and close a page for each request
keep the page open across requests

My solution is a "hack" that will need to be updated every time a new release of Crawlee updates the .

Ffull-green @Marc Plouhinec interesting topic, in my case I created two crawlers one with Pu...

unexpected-ivory•3/5/24, 3:09 AM

Do you have a code sample that you could possibly share how you sync cookies and other context from Puppeteer Crawler to the Brasic crawler?

Uunexpected-ivory Do you have a code sample that you could possibly share how you sync cookies and...

unique-blueOP•3/5/24, 3:19 AM

Sorry I don't share cookies: I open the website, I run fetch as much as I can, and when I get blocked I re-open a new browser page with a fresh session

MDN Web Docs

Fetch API - Web APIs | MDN

The Fetch API provides an interface for fetching resources (including across the network). It is a more powerful and flexible replacement for XMLHttpRequest.