unwilling-turquoise
unwilling-turquoise•2y ago

Need help with page concurrency

I am working on booking automation app, where I want to concurrently process pages and swift through offers and accept the ones which fits in the requirements and rejects others. is there any way I can integrate crawlee and use its concurency power in my program? here's my code: 
class PageManager
{
    constructor()
    {
        this.validator = new Validator()
    }

    async acceptOffers(offers, pageNavigator, newInjectedPage, pageSettings) 
    {
        for (let offer of offers)
         {
          const isAcceptable = this.validator.checkRequirements(offer)
          if (isAcceptable)
           {
              const page = newInjectedPage(pageSettings)
              const navigator = new pageNavigator()
              await navigator.acceptOffer(offer.hash)
              await navigator.closePage()
              Logger.log(isAcceptable, offer)
           } else {
            Logger.log(isAcceptable, offer)
              }
}
        
    }

}
class PageManager
{
    constructor()
    {
        this.validator = new Validator()
    }

    async acceptOffers(offers, pageNavigator, newInjectedPage, pageSettings) 
    {
        for (let offer of offers)
         {
          const isAcceptable = this.validator.checkRequirements(offer)
          if (isAcceptable)
           {
              const page = newInjectedPage(pageSettings)
              const navigator = new pageNavigator()
              await navigator.acceptOffer(offer.hash)
              await navigator.closePage()
              Logger.log(isAcceptable, offer)
           } else {
            Logger.log(isAcceptable, offer)
              }
}
        
    }

}
problem with this one is, if in some page errors like page loading, booking no longer available occur, it will lose the opportunity to accept other potential offers. So, I want it to process simultaneously/concurrently so in case of errors, it doesn't miss other offers. little help is appreciated. Thanks
1 Reply
harsh-harlequin
harsh-harlequin•2y ago
Sure there is an example: https://crawlee.dev/docs/3.5/examples/crawl-multiple-urls It will crawl the URLs concurrently, it will also handle the page errors and retry the requests automatically You can add your code inside the requestHandler function You can find more info in the docs: https://crawlee.dev/docs/3.5/quick-start

Did you find this page helpful?