Stop scraping in the middle of a route handler if some condition is met?

hey folks, basically the title, I want to stop scraping in the middle of a route handler if some conditon is met because the full function is a little expensive computationally.

I am using

return;

return;

to exit the route handler when some condition is met but I am facing issues with the the node process hanging indefinitely after scraping is done. I had a previous thread using

crawler.teardown()

crawler.teardown()

and how removing the return statement stopped the hanging issue.

But even in handlers where I am not calling

crawler.teardow()

crawler.teardow()

, I am facing hanging issues.

is there a better way to accomplish this?

Lukas Krivka•12/7/23, 1:00 PM

Hanging process has nothing to do with

return

return

. Yes, having if statements with return is the way to go

Cclean-aquamarine Hanging process has nothing to do with `return`. Yes, having if statements with ...

verbal-limeOP•12/7/23, 3:29 PM

can you some ideaa about how I can go about debugging this? or any potential fixes I can possibly implement. would appreciate any help you can give

Lukas Krivka•12/7/23, 3:39 PM

Does it hang after the crawler finishes? Why would you use teardown and not let it process the requests till the end?

Cclean-aquamarine Does it hang after the crawler finishes? Why would you use teardown and not let ...

verbal-limeOP•12/7/23, 10:37 PM

no, the teardown usecase was different, I had to open a couple of pages in the route itself to grab info and wanted to skip it to avoid spamming the site if possible and also to stop the crawler asap but our main use case was to have some kind of "resume" for it.

verbal-limeOP•12/7/23, 10:38 PM

in the current use case I'm not calling teardown anywhere

verbal-limeOP•12/7/23, 10:38 PM

it's just a simple route and the crawler hangs after it logs the final request statistics

Cclean-aquamarine Does it hang after the crawler finishes? Why would you use teardown and not let ...

verbal-limeOP•12/7/23, 11:00 PM

and this is the route handler(https://pastebin.com/8JaY9PUR), the crawler's startup url only hits this route and is supposed to shut down but it hangs indefinitely

verbal-limeOP•12/7/23, 11:01 PM

and its stuck here

verbal-limeOP•12/8/23, 3:55 AM

nvm its not crawlee

verbal-limeOP•12/8/23, 3:55 AM

its mongoose screwing around