Do you know a dashboard for Crawleee?

At a glance

The community member is looking for a dashboard to monitor their Crawlee crawlers. They found the Crawlab platform, but are interested in other options. A community member suggests the Apify platform, which has seamless support for Crawlee, but the original poster cannot use a cloud solution due to sensitive data. The community members then discuss running the crawlers locally on a cluster of Raspberry Pis at the poster's home. To create a custom dashboard, the community members recommend setting up a server on one of the Raspberry Pis to act as an API for storing data about the crawlers, and then building a UI service to interact with the API.

Useful resources

LLeMoussel

I want to monitor all Crawlee crawlers. To do this I looked for a dashboard to control the crawlers.
I only found this distributed web crawler management platform: Crawlab (https://github.com/crawlab-team/crawlab).

Do you know of any others?

9 comments

tthek1tten

Have you tried the Apify platform? If you deploy your crawlers to the platform, you can run them and monitor them right from the dashboard there!

https://apify.com

The great thing about this solution is that Crawlee was built with the Apify platform in mind, so there is seamless support

LLeMoussel

Thanks for your answer, but using sensitive data, I cannot use a Cloud solution. That's why I'm looking for an open source solution and I use Crawlee.

tthek1tten

So, your crawlers will only be run locally?

LLeMoussel

Yes, I have a cluster of server where crawlee crawler run.

tthek1tten

How are these servers being hosted?

LLeMoussel

These are Raspberry pi's at my house

AApifyBot

just advanced to level 1! Thanks for your contributions! 🎉

tthek1tten

In order to make a dashboard, you'd need a centralized location where you could store data about each crawler. A SQL or Mongo database would be great for this.

You could have a server hosted on one of your rasp pis that serves as an API for storing data about crawlers. When a new crawler is started, a new row could be added to the "crawlers" table by sending a request from the crawler to your API.

Then, you could create some sort of UI service with Electron or React Native that uses your API.

tthek1tten

With this custom solution, you could make the dashboard as minimalistic or as robust as you like. You'd have full control. And everything would be hosted locally.

Add a reply

Apify Discord Mirror

Do you know a dashboard for Crawleee?