Resurect an Actor run
I develop an actor that runs on Apify platform. The actor will basically take a list of URL in the INPUT and open each one of them. Simple.
I run the Actor and it timed out. Fine. Then I resurrected it and strangely enough, it started loading the INPUT list of URL from the begining.
Isn't it strange ?
7 Replies
You need to persist the progress in some way for example, by using a RequestQueue, a Key-Value Store (KVS), your own external database on a VPS, or even by extracting logs (e.g., if you log something like Processed page 10/500, although I wouldn’t recommend relying on logs since that approach is rather fragile), etc.
The Apify Dataset is not enough ?
And if I understand you correctly, when you say "persist the progress in some way" it means that by default, resurrecting will start from scratch ?
I need to handle the specific case of resurection in my actors ?
Exactly you need to persist the progress somehow, because Apify doesn’t do that automatically. You have to handle it yourself, whether the user aborts, a timeout occurs, or Apify internally migrates the run to another server for some reason. That’s why, before starting a run, you should check the progress using whichever option you’ve chosen.
Tks for your answer. very clear !
Are there any Actor implementation example somewhere ?
@anchor (guillim) just advanced to level 4! Thanks for your contributions! 🎉
(that you would be aware of)
yeah, for handling URLs you should use requestQueue
And for saving some important data You can use useState():
https://crawlee.dev/js/api/next/core/function/useState
Also see this article for inspiration:
https://docs.apify.com/academy/expert-scraping-with-apify/solutions/handling-migrations