Aaccused-moccasin
High Volume Scraping
I'm evaluating Crawlee for my startup that will require us to scrape several hundred websites. These sites are non-ecommerce nor social media and require interaction with the page (feeding in a list of search parameters, clicking submit buttons, etc.) The documentation seems to imply that I need to use a headless browser in order to interact with the site, but headless browsers consume tons of memory when compared to the non browser counterparts and are overkill for sites that do not render Javascript. Other tools seem to have workarounds for these sites such as adding the pageviewstate in the crawler with parameters taken from the site. Does Crawlee have options to interact with pages without using the browser object?
This is the official developer community of Apify and Crawlee.
13,923Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
