Scraping single page with load more button
Hi, I just discovered Crawlee and seems a very great project.
I'm scraping a single url (https://jobs.workable.com/search) that contains a list of items with a load more button. Each time an item is clicked a floating modal show the item information.
In this scenario all the power of crawlee to remember visited urls, retries, etc is not a help.
My idea is:
- From the start page, click on each of the initial items and scrape its content
- Click on the load more button and repeat the process.
The help I'm requesting is in how to apply best practices for:
- how to "remember/store" the last scrapped item index/id
- how to handle with errors
Thanks in advance
I'm scraping a single url (https://jobs.workable.com/search) that contains a list of items with a load more button. Each time an item is clicked a floating modal show the item information.
In this scenario all the power of crawlee to remember visited urls, retries, etc is not a help.
My idea is:
- From the start page, click on each of the initial items and scrape its content
- Click on the load more button and repeat the process.
The help I'm requesting is in how to apply best practices for:
- how to "remember/store" the last scrapped item index/id
- how to handle with errors
Thanks in advance
Search jobs using the new Job Finder from Workable. Explore thousands of open job listings hosted by Workable‘s all-in-one recruitment software, trusted by companies worldwide.
