eastern-cyan
eastern-cyan•2y ago

isFinishedFunction, check other crawler?

Hello, two questions. Is there a way to call this.isFinishedFunction so it calls the original function but also just add another web_crawler_queue isFinished on top of it? the uncommented out function I tried, worked somewhat, but, after a long running web_crawler_queue finished, it just kept giving me a stalled error, and, this crawler that this function belongs to never finished. 
    autoscaledPoolOptions: {
      isFinishedFunction: async () => {
        const web_crawler_queue = await RequestQueue.open(place_id)
//        return this.isFinishedFunction() && await web_crawler_queue.isFinished()
        return await request_queue.isFinished() && await web_crawler_queue.isFinished()

      }
    },
    autoscaledPoolOptions: {
      isFinishedFunction: async () => {
        const web_crawler_queue = await RequestQueue.open(place_id)
//        return this.isFinishedFunction() && await web_crawler_queue.isFinished()
        return await request_queue.isFinished() && await web_crawler_queue.isFinished()

      }
    },
3 Replies
MEE6
MEE6•2y ago
@bmax just advanced to level 3! Thanks for your contributions! 🎉
Lukas Krivka
Lukas Krivka•2y ago
There is no original function since you are putting your own there in options. If you would want to use original, you would have to monkeypatch it after defining. 
// not sure about the .bind
const origFn = crawler.autoscaledPool.isFinishedFunction.bind(crawler)
crawler.autoscaledPool.isFinishedFunction = async () => {
    const orig = origFn()
    // etc.
}
// not sure about the .bind
const origFn = crawler.autoscaledPool.isFinishedFunction.bind(crawler)
crawler.autoscaledPool.isFinishedFunction = async () => {
    const orig = origFn()
    // etc.
}
eastern-cyan
eastern-cyanOP•2y ago
got it, will try this my code above almost works @Lukas Krivka but for some reason doesn't all the time.. which is basically is my current request_queue finished and is my other crawler finished but sometimes it doesn't pick up and I'll get an log along the lines of : "x, y, z ahs been stalled for 350 seconds" 
{
  "isSystemIdle": false,
  "memInfo": {
    "isOverloaded": true,
    "limitRatio": 0.2,
    "actualRatio": 1
  },
  "eventLoopInfo": {
    "isOverloaded": false,
    "limitRatio": 0.6,
    "actualRatio": 0.057
  },
  "cpuInfo": {
    "isOverloaded": true,
    "limitRatio": 0.4,
    "actualRatio": 1
  },
  "clientInfo": {
    "isOverloaded": false,
    "limitRatio": 0.3,
    "actualRatio": 0
  }
}
{
  "isSystemIdle": false,
  "memInfo": {
    "isOverloaded": true,
    "limitRatio": 0.2,
    "actualRatio": 1
  },
  "eventLoopInfo": {
    "isOverloaded": false,
    "limitRatio": 0.6,
    "actualRatio": 0.057
  },
  "cpuInfo": {
    "isOverloaded": true,
    "limitRatio": 0.4,
    "actualRatio": 1
  },
  "clientInfo": {
    "isOverloaded": false,
    "limitRatio": 0.3,
    "actualRatio": 0
  }
}
this: RNING","msg":"RequestQueue: The request queue seems to be stuck for 370s, resetting internal state.","inProgress":[]}

Did you find this page helpful?