Crawlee's Compatibility with Kivy

Hey all,
I was wondering if there are any specific compatibility issues with using crawlee within a kivy app, because whenever I run the script attached to grab the name of a product in amazon, it gives a recursion error that stems within the default_handler, specifically when the url is pulled from the context.

FWIW, the UI pipeline is App -> ScreenManager -> Main Screen & Second Screen, wherein the main screen transitions to the second screen via switch_to()

unfortunately this is one of those problems that are very out of my depth, so if i'm missing some crucial information please let me know. also there's a lot of imports that are technically redundant, but that's mostly because the script is a trimmed down version of the main file

recursionErrorFile.py7.97KB

Solution

Hey, @Glitchy_mess

Try adding

configure_logging=False

configure_logging=False

when initializing the crawler.

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

Jump to solution

Exp•8/15/25, 5:46 PM

I think likely event-loop conflict — run Crawlee in a separate thread/process from Kivy’s main loop to avoid recursion errors when switching screens.

Glitchy_messOP•8/15/25, 6:28 PM

so fun thing about that, earlier i did try seeing if a LLM could visualize a solution so that i had something to go off of, and gpt had the same idea of using threading, but that also ran into the same issue of being stuck in recursion hell which makes me think there's something else that's off?
I'll admit that this is one of those things i just haven't looked into before so its possible the stuff it wrote for the threading is just horrendously wrong, and if so i'd super appreciate pointers in the right direction for this

gptThreading.py16.03KB

Exp•8/15/25, 7:09 PM

did you try to use multi-threading?

Glitchy_messOP•8/15/25, 7:20 PM

haven't tried that, are you thinking something like running the second screen and the crawlee stuff on different threads when it comes time to transition?

Exp•8/15/25, 7:21 PM

Yes

Glitchy_messOP•8/15/25, 7:39 PM

sadly it still hits the recursion error with this change

Glitchy_messOP•8/15/25, 7:40 PM

(mainRunner() just has the class instance and the async.run call)

Glitchy_messOP•8/15/25, 8:05 PM

fwiw i also tried bringing the crawlee stuff outside of all of the kivy things, but even that seems to throw the recursion error so idk what kivy is doing to make crawlee collapse this hard

multiThreadingExperiment.py8.46KB

MEE6•8/15/25, 8:05 PM

@Glitchy_mess just advanced to level 2! Thanks for your contributions!

Solution

Mantisus•8/17/25, 11:34 PM

Hey, @Glitchy_mess

Try adding

configure_logging=False

configure_logging=False

when initializing the crawler.

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

crawler = PlaywrightCrawler(
                        use_session_pool=True, 
                        max_session_rotations=0,
                        browser_pool=browserPoolVar,
                        session_pool=SessionPool(max_pool_size=1),
                        configure_logging=False
                        )

MMantisus Hey, @Glitchy_mess Try adding `configure_logging=False` when initializing the ...

Glitchy_messOP•8/18/25, 5:17 AM

holy smokes that fixed it
i'm so curious on how configure_logging ended up being the issue since the param just says that it'll set up the logging infra automatically? like is it just a clash in how crawlee and kivy do events?

Glitchy_messOP•8/18/25, 5:17 AM

that said, super appreciated, thank you both so much!!! (like even if threading didn't get places, i got to learn a little bit of how to actually impliment it which is neat)

Mantisus•8/18/25, 9:16 AM

like is it just a clash in how crawlee and kivy do events?

To be honest, I didn't look into the details. The solution was found because of the desire to get a normal log of the problem; any modification of the log output removed the error.

I can assume that this is because redefines and modifies the base logger. And , with , clears all existing loggers. I think this is what creates the conflict.