Hhurt-tomato
infinite scrolling - best practice
I'm crawling a products page on an online toy store. There is an 'all products' page that loads 20 products at a time. It's an infinite scrolling scenario so no pagination buttons. I followed the tutorial from the blog and added the await for netowrkidle state and infinite_scroll method to implement the page to be scrolled continuously. After each scroll, 20 more products are loaded and the process repeats. The issue I'm facing is that there are thousands of products to retrieve. So, eventually a timeout error is thrown.
To me, this approach seems rather inefficient. Not only is the timeout an issue (which I'm sure can be adjusted), but since it might take an hour to scroll all products any random event (popup, or some anti-scrape challenge) might appear and end the run before the enqueued product detail links can even be processed.
I get that using API requests, where possible is preferable but I haven't identified those yet. I do see that after each scroll, the URL is appended with ?page=2. So scroll down again and it become ?page=3, etc.
Would would be best practice to implement this cleanly within the Crawlee framework? Any pointers from you gurus? Thanks!
Below is the default handler implementing the infinite scroll. I'm enqueuing all the product detail links which are then processed by a separate route handler.
@router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'default_handler is processing: {context.request.url}')
await context.page.wait_for_selector('.ais-InfiniteHits-list')
# Infinite scrolling: wait for page load to complete and scroll to bottom
await context.page.wait_for_load_state('networkidle')
await context.infinite_scroll()
await context.enqueue_links(
selector='.ais-InfiniteHits-item .result-wrapper div a',
label='PRODUCT',
unique_key=context.request.url
)
To me, this approach seems rather inefficient. Not only is the timeout an issue (which I'm sure can be adjusted), but since it might take an hour to scroll all products any random event (popup, or some anti-scrape challenge) might appear and end the run before the enqueued product detail links can even be processed.
I get that using API requests, where possible is preferable but I haven't identified those yet. I do see that after each scroll, the URL is appended with ?page=2. So scroll down again and it become ?page=3, etc.
Would would be best practice to implement this cleanly within the Crawlee framework? Any pointers from you gurus? Thanks!
Below is the default handler implementing the infinite scroll. I'm enqueuing all the product detail links which are then processed by a separate route handler.
@router.default_handler
async def default_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'default_handler is processing: {context.request.url}')
await context.page.wait_for_selector('.ais-InfiniteHits-list')
# Infinite scrolling: wait for page load to complete and scroll to bottom
await context.page.wait_for_load_state('networkidle')
await context.infinite_scroll()
await context.enqueue_links(
selector='.ais-InfiniteHits-item .result-wrapper div a',
label='PRODUCT',
unique_key=context.request.url
)
