rare-sapphire•3y ago
RequestQueue.open issue in dockerized app
When i'm trying to run my app with crawlee inside docker container (from apify/actor-node) "RequestQueue.open()" code just infinite executed without any errors and hinders further program execution. During usual execution at my PC it works fine.
13 Replies
conscious-sapphire•3y ago
Could you provide some repro? Stack trace? Something? I'd say we never experienced anything like that. Also - which crawlee version are you using?
rare-sapphireOP•3y ago
You could see simplified code here https://github.com/RDzhalmukhamedov/docker-crawlee-test. I use actual version of crawlee: 3.3.2.
conscious-sapphire•3y ago
I had a very briefly look, so might have misread it, but are you creating crawler instance first and then assigning the queue to it?
rare-sapphireOP•3y ago
Yes, does it matter? When I do not declare the queue separately, but immediately pass the request with link to crawler.run(), the program does not work in the same way. Just freezes on the crawler.run() method
conscious-sapphire•3y ago
The thing is - if you don't explicitly pass the queue in crawler constructor - it silently creates the queue. It simply is not how it's designed, so I am not really sure what's happening in this case. This does not explain why it hangs on crawler.run() though
rare-sapphireOP•3y ago
Ok, got it. I could update project with immediate passing case
@mrkrouly just advanced to level 1! Thanks for your contributions! 🎉
conscious-sapphire•3y ago
yeah, could you please give it a try, and if it's still the same (but I guess it will still since you mentioned you already tried it this way) - I will pass it to the team together with repro
rare-sapphireOP•3y ago
Tried one more time. Still the same behavior. Updated project with immediate passing and little readme
We need to know more about your platform since it is obviously connected to that, not the bug in the code
rare-sapphireOP•3y ago
I tried to run it locally (Win 10 Enterprise Version 22H2, Build 19045.2965; Docker version 23.0.5, build bc4487a; Docker Compose version v2.17.3) and also tried to run at Fly.io (https://fly.io/). Sadly, I had same behavior.
sunny-green•3y ago
Hey @mrkrouly I've just tested out the repo locally and haven't been able to reproduce the issue!
Can you try building your image without having a storage folder?
That said it should work with it..
Found the issue
Now to figure out how to fix it...
Fix figured out! Will PR :D
thanks for reporting
Reminder
as a temporary fix, make sure you have no
storage folder when invoking docker build . (or docker buildx <...>. that will also fix the issuerare-sapphireOP•3y ago
Thanks for help!