xenial-black•2y ago
Stop scraping in the middle of a route handler if some condition is met?
hey folks, basically the title, I want to stop scraping in the middle of a route handler if some conditon is met because the full function is a little expensive computationally.
I am using
return; to exit the route handler when some condition is met but I am facing issues with the the node process hanging indefinitely after scraping is done. I had a previous thread using crawler.teardown() and how removing the return statement stopped the hanging issue.
But even in handlers where I am not calling crawler.teardow(), I am facing hanging issues.
is there a better way to accomplish this?6 Replies
Hanging process has nothing to do with
return. Yes, having if statements with return is the way to goxenial-blackOP•2y ago
can you some ideaa about how I can go about debugging this? or any potential fixes I can possibly implement. would appreciate any help you can give
Does it hang after the crawler finishes? Why would you use teardown and not let it process the requests till the end?
xenial-blackOP•2y ago
no, the teardown usecase was different, I had to open a couple of pages in the route itself to grab info and wanted to skip it to avoid spamming the site if possible and also to stop the crawler asap but our main use case was to have some kind of "resume" for it.
in the current use case I'm not calling teardown anywhere
it's just a simple route and the crawler hangs after it logs the final request statistics
and this is the route handler(https://pastebin.com/8JaY9PUR), the crawler's startup url only hits this route and is supposed to shut down but it hangs indefinitely
xenial-blackOP•2y ago
and its stuck here
xenial-blackOP•2y ago
nvm its not crawlee
its mongoose screwing around