How to add multiple crawler in the same desktop program...

At a glance

The community member is using crawlee and electron to develop a desktop program and has two problems: 1) they want to start multiple crawler tasks at the same time with independent parameter settings, but couldn't find information about this in the documentation, and 2) they want to add pause and restart features to the crawler task, but couldn't find the related functions in the documentation.

In the comments, another community member provided some suggestions. They mentioned that the community member could use the autoscaledPool property to pause and resume the crawler, or use abort to stop the crawler and resurrect to restart it. However, the community member clarified that they are using crawlee, not Apify, so they are not sure if those methods would work in their case.

Another community member then provided an alternative solution, suggesting that the community member could create named routes and pass the parameter settings using the userData object, or create multiple crawlers but be aware of potential issues with persistence.

There is no explicitly marked answer in the comments.

Useful resources

hhunterleung.

Hello everyone, I am using crawlee and electron to develop a desktop program, I met two problems:

I want to start multiple crawler tasks at the same time in my software, and each crawler task can have independent parameter settings.

I don't see anything about this in the documentation. How should I implement this requirement?

I wanna add pause and restart features to the crawler task,but I haven't seen the relate function in the documentation.

Can someone give me some tip? I would be very grateful.

4 comments

HHamza

Here is the documentation for running tasks: https://docs.apify.com/api/v2#tag/Actor-tasksRun-collection/operation/actorTask_runs_post

You can use the autoscaledPool property to pause and resume the crawler: https://crawlee.dev/api/core/class/AutoscaledPool#pause, https://crawlee.dev/api/core/class/AutoscaledPool#resume,

or you can use abort to stop the crawler: https://docs.apify.com/api/client/js/reference/class/RunClient#abort, and use resurrect to resurrect it: https://docs.apify.com/api/client/js/reference/class/RunClient#resurrect

hhunterleung.

hi, Hamza .Thanks for your help. I am using crawlee but not apify in my project . so I don't know these method you mentioned if works in my case.

https://docs.apify.com/api/v2#tag/Actor-tasksRun-collection/operation/actorTask_runs_post
https://docs.apify.com/api/client/js/reference/class/RunClient#abort
https://docs.apify.com /api/client/js/reference/class/RunClient#resurrect

AApifyBot

just advanced to level 2! Thanks for your contributions! 🎉

HHamza

Hay ,

My bad for not understanding the question.

1- You can do it by creating named routes: https://crawlee.dev/docs/introduction/refactoring#routing, pass the parameter settings using userData: https://crawlee.dev/api/core/class/Request#userData object to each route label

Second option is you technically can create multiple crawlers but the persistence will probably get messed up since it is indexed by order when it started (e.g. SDK_CRAWLER_STATISTICS_0), and you would need to not use a queue or use named queue

2- You can use the pause and resume as mentioned above.

Add a reply

Apify Discord Mirror

How to add multiple crawler in the same desktop program? thanks