Is there way to store the state and continue?
Hello there, well, I am looking for a way to store current state that where is crawler is crawling, and if anything happen and error occured and crash, then we need to fix it and continue from there.
For example, I wrote a program that crawls google's search page. And I want to crawl 1000+ more page, thus that should take a loooong time.
While crawling, there was error occured due to our program's problem, like we missed special button of google's page.
We fixed it now, but we have no way to continue from there, but we want to, because it already took like 5 hours and we have to spend more waste 5 hours.
For example, I wrote a program that crawls google's search page. And I want to crawl 1000+ more page, thus that should take a loooong time.
While crawling, there was error occured due to our program's problem, like we missed special button of google's page.
We fixed it now, but we have no way to continue from there, but we want to, because it already took like 5 hours and we have to spend more waste 5 hours.
