absent-sapphire•3y ago
Is it preferable to keep a crawler(s) alive or keep spawning and tearing them down?
Hi, I am using puppeteer crawler to crawl my site to take snapshots for a visual regression testing system. There is a CI pipeline that will add a bunch of urls to crawl. Is there any advantage of just keeping a crawler running and waiting for requests to enter the queue?
1 Reply
robust-apricot•3y ago
I would suggest not closing the crawler and only add new requests to the queue. This way, there will be no cold start needed for the browser initialization.