Apify & Crawlee

This is the official developer community of Apify and Crawlee.

Join

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

Join

crawlee-js

apify-platform

crawlee-python

💻hire-freelancers

🚀actor-promotion

💫feature-request

💻creators-and-apify

🗣general-chat

🎁giveaways

programming-memes

🌐apify-announcements

🕷crawlee-announcements

👥community

sunny-green

12/2/2023

Apify not executing scrapy's close_spider function and going insane after it finishes scraping.

Hello, I got a little problem. As I said in the title, my script does not execute the close_spider function and when the scraping finishes, it goes in an infinite loops. I guess that's why close_spider doesn't get executed. Can anyone help?

other-emerald

12/1/2023

Using Poetry

What would be a recommended way to use Poetry to install dependencies instead of pure pip? Anyone has this working in their Apify Dockerfile?

plain-purple

11/30/2023

Scraping Bet365's live odds

Hey everyone, I am trying to scrape data from games that are in-play on Bet365. Unfortunately, their site blocks all requests via Cloudflare, and connecting to the websockets without getting data from the site first is impossible. I would also prefer not to use Selenium as it is hard to scale and the data is prone to errors. They are also are able to detect Selenium and seem to block most typical workarounds, so it doesn't really even seem possible lol. Any help would be greatly appreciated.

optimistic-gold

11/22/2023

How to scrape data from website using websocket in real time(Python)?

I have to scrape data from sports site in real time. The match results are updated on website in real time. I have to get updated data in real time using Python and before that, I have to sign in that website. And it has captcha authentication. How can I solve this problem?...

rival-black

11/22/2023

API CLIENT for Python doesn’t open on the website

When going to any actor, then clicking on “API” dropdown -> “API Client” Node.js and cURL work fine but the site freezes when pressing on python. I have same problem with all actors I tried...

absent-sapphire

11/13/2023

ConnectionPool.__init__() got an unexpected keyword argument 'socket_options'

Hey guys, I get this error when trying to init the apify client. Appreciate your help... Thanks...

like-gold

11/4/2023

How to export size?

Url: https://www.fuel.com.gr/el/catalogsearch/result/index/?q=Dunk Every shoes is under a li class And shoe size in div class = swatch-option-text Do you know how can I export the value using requests?...