Instance not refreshing in API

I am building an API that crawls websites using crawlee's playwright crawler.

Once the request goes to API endpoint to crawl, it crawls the website successfully but whenever new request with different details is made, It denies to crawl again.

static-pinkOP•12/29/23, 1:50 AM

Basically once the crawling is completed, its instance is not getting shut down and able to restart again

dead-brown•1/3/24, 3:30 AM

@Kunal Verma Did you find a solution for this problem? I'm facing the same issue.

static-pinkOP•1/3/24, 4:42 AM

No, now trying to maintain a queue for requests and after each run dropping requestQueue

dead-brown•1/3/24, 6:32 AM

Ok, I did the same. Created named request queue and dropping it after crawl.

How can we limit the max urls to crawl per requestQueue? This is not the same as maxRequestsPerCrawl.