sunny-green•2y ago
Scrapy integration silently throws away redirects
Cheers, I just noticed that my custom scraper makes quite different number of requests locally and through Apify, while the code, URLs, parameters, everything is the same. The same Scrapy spider produces 720 items locally, but 370 through Apify
Anyone has any clue what could be the root cause, where to look? Just from the logs I can't see anything. The only clue I noticed is that on Apify the scraper makes no POST requests, but that probably isn't enough to debug the root cause 🤔
Is there a way I can raise logging or something on Apify? How can I best approach this? How to debug this?
4 Replies
sunny-greenOP•2y ago
Running with or without proxy makes no difference
sunny-greenOP•2y ago
After much debugging, I filed https://github.com/apify/apify-sdk-python/issues/185
GitHub
Scrapy integration silently throws away redirects · Issue #185 · ap...
I'm debugging a situation where the same Scrapy spider produces 720 items locally, but 370 through Apify. After a day of detective work I figured out that the Scrapy-Apify integration probably ...
Hi Honza, thanks for the report, I'll try to look into it soon
sunny-greenOP•2y ago
Thanks! 🙇♂️