Oordinary-sapphire
Serialize crawlerContext and response objects
With PHP you can easily serialize objects and later deserialize them.
I'm wondering if it's possible to serialize the crawlerContext and response objects and then save to file and later deserialize it to parse the results. The reason I'd like to do this to have one crawler that goes through the site and saves all responses to file.
After everything is crawled I have dozens of nodejs pods that go through all the crawlerContext and responses and parse them.
I'm wondering if it's possible to serialize the crawlerContext and response objects and then save to file and later deserialize it to parse the results. The reason I'd like to do this to have one crawler that goes through the site and saves all responses to file.
After everything is crawled I have dozens of nodejs pods that go through all the crawlerContext and responses and parse them.
