ambitious-aqua•3y ago

PlaywrightCrawler.requestHandler: Error: mouse.move: Target page, context or browser has been closed

In the PlaywrightCrawler.requestHandler I calling page.mouse.move and sometimes I get this error: mouse.move: Target page, context or browser has been closed Here the sequence of calls:

async requestHandler( {request, response, page, enqueueLinks, log, proxyInfo} )
{
    ...
    await sleep( interval );
    await page.mouse.move( rnd(100,400), rnd(40,300) );
    await sleep( interval );
    ...
    content = await page.content();
}

async requestHandler( {request, response, page, enqueueLinks, log, proxyInfo} )
{
    ...
    await sleep( interval );
    await page.mouse.move( rnd(100,400), rnd(40,300) );
    await sleep( interval );
    ...
    content = await page.content();
}

In case I catch the exception thrown in page.mouse.move and contine - than I get almost the same thing when calling page.content(): page.content: Target page, context or browser has been closed I would like to move mouse randomly - I think, the make my scraper "human-like". But something is going wrong here and I can not figure out what. Sometimes this code works and sometimes I see these errors! Pls help! UPDATE: and sometimes the error message is: ERROR requestHandler: Request failed and reached maximum retries. page.goto: Navigation failed because page was closed! I think, somehow these error messages are related...

4 Replies

Alexey Udovydchenko•3y ago

Page might detect bot and close itself, as a quick solution try Playwright (its less detectable)

Lukas Krivka•3y ago

page close usually means it just crashed because your CPU or memory was overloaded. Try to decrease concurrency or increase system resources

ambitious-aquaOP•3y ago

I do not think this is my case (minConcurrency=maxConcurrency=4, i do not think this is a lot) but I check this again.

CPU or memory was overloaded.

with CPU - this is clear. with RAM - do you think this is "not enough RAM for +1 browser" or "not enough RAM for the node process"? What is the best way to detect the shortage of RAM ?

Lukas Krivka•3y ago

I mean the browser, it depends a lot on how big the page is. Just try to reduce concurrency to 1 and see if you still see the crashes. Also, don't set minConcurrency, because it cannot downscale then, use desiredConcurrency of autoscaledPoolOptions

PlaywrightCrawler.requestHandler: Error: mouse.move: Target page, context or browser has been closed

Did you find this page helpful?