constant-blue•3y ago
Download Delay
does crawlee support download delay? Like in Scrapy? Because I want to crawl a website but this website has delay before to load its content, so my current crawlee project didn't get the content of the website.
5 Replies
hey, how about waiting for a selector?
constant-blueOP•3y ago
I will scrape many and different website's so that why I'm not depending on a selector
maxRequestsPerMinute
Use waitUntil() (https://crawlee.dev/api/playwright-crawler/namespace/playwrightUtils#waitUntil) Just find proper event to wait for.
Or You can use sleep() (https://crawlee.dev/api/utils/function/sleep) (not the best approach).
constant-blueOP•3y ago
thanks for this, I tried both of these and set a usecase for each. Also I try the networkIdle