Glitchy_mess•3w ago

Crawlee's Compatibility with Kivy

Hey all, I was wondering if there are any specific compatibility issues with using crawlee within a kivy app, because whenever I run the script attached to grab the name of a product in amazon, it gives a recursion error that stems within the default_handler, specifically when the url is pulled from the context. FWIW, the UI pipeline is App -> ScreenManager -> Main Screen & Second Screen, wherein the main screen transitions to the second screen via switch_to() unfortunately this is one of those problems that are very out of my depth, so if i'm missing some crucial information please let me know. also there's a lot of imports that are technically redundant, but that's mostly because the script is a trimmed down version of the main file

recursionErrorFile.p...

Solution:

Hey, @Glitchy_mess Try adding configure_logging=False when initializing the crawler. ```python...

Jump to solution

13 Replies

Exp•3w ago

I think likely event-loop conflict — run Crawlee in a separate thread/process from Kivy’s main loop to avoid recursion errors when switching screens.

Glitchy_messOP•3w ago

so fun thing about that, earlier i did try seeing if a LLM could visualize a solution so that i had something to go off of, and gpt had the same idea of using threading, but that also ran into the same issue of being stuck in recursion hell which makes me think there's something else that's off? I'll admit that this is one of those things i just haven't looked into before so its possible the stuff it wrote for the threading is just horrendously wrong, and if so i'd super appreciate pointers in the right direction for this

gptThreading.py

Exp•3w ago

did you try to use multi-threading?

Glitchy_messOP•3w ago

haven't tried that, are you thinking something like running the second screen and the crawlee stuff on different threads when it comes time to transition?

Exp•3w ago

Yes

Glitchy_messOP•3w ago

sadly it still hits the recursion error with this change

Glitchy_messOP•3w ago

(mainRunner() just has the class instance and the async.run call)

Glitchy_messOP•3w ago

fwiw i also tried bringing the crawlee stuff outside of all of the kivy things, but even that seems to throw the recursion error so idk what kivy is doing to make crawlee collapse this hard

multiThreadingExperi...

MEE6•3w ago

@Glitchy_mess just advanced to level 2! Thanks for your contributions! 🎉

Solution

Mantisus•2w ago

Hey, @Glitchy_mess Try adding configure_logging=False when initializing the crawler.

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

Glitchy_messOP•2w ago

holy smokes that fixed it i'm so curious on how configure_logging ended up being the issue since the param just says that it'll set up the logging infra automatically? like is it just a clash in how crawlee and kivy do events?

Glitchy_messOP•2w ago

that said, super appreciated, thank you both so much!!! (like even if threading didn't get places, i got to learn a little bit of how to actually impliment it which is neat)

Mantisus•2w ago

like is it just a clash in how crawlee and kivy do events?

To be honest, I didn't look into the details. The solution was found because of the desire to get a normal log of the problem; any modification of the log output removed the error. I can assume that this is because kivy redefines and modifies the base logger. And crawlee, with configure_logging=True, clears all existing loggers. I think this is what creates the conflict.

Crawlee's Compatibility with Kivy

Did you find this page helpful?