Eefficient-indigo
Crawlee stops after about 30 items pushed to the datastore, repeats the same data on next run.
I'm writing my first Actor using Crawlee and Playwright crawler to scrape website https://sreality.cz.
I wrote a crawler using as much as possible from the examples in the documentation. It works like this:
1. Start on the first page of search, for example this one.
2. Skip ad dialog, if it shows.
3. Find all links to next pages and add them to the queue with
4. Find all links to individual items (apartments, houses, whatever) and add them to the queue with
5. If next page to process is an item page, scrape the data and save with
In theory, this is all I need to scrape the entire search result list. However what I experience is that it will enqueue all the links (around 185) but only process around 30 of them before finishing. Very strange.
I tried to set
Maybe I'm missing something but I don't see why it would just stop after around 30 pages. Is there another config somewhere that controls this?
Even more strange, it then logs the final statistic where it says something like
I wrote a crawler using as much as possible from the examples in the documentation. It works like this:
1. Start on the first page of search, for example this one.
2. Skip ad dialog, if it shows.
3. Find all links to next pages and add them to the queue with
enqueueLinks()enqueueLinks().4. Find all links to individual items (apartments, houses, whatever) and add them to the queue with
enqueueLinks()enqueueLinks().5. If next page to process is an item page, scrape the data and save with
pushData()pushData(). Otherwise, if it's another page, repeat from 3.In theory, this is all I need to scrape the entire search result list. However what I experience is that it will enqueue all the links (around 185) but only process around 30 of them before finishing. Very strange.
I tried to set
maxRequestsPerCrawl: 1000maxRequestsPerCrawl: 1000, didn't help.Maybe I'm missing something but I don't see why it would just stop after around 30 pages. Is there another config somewhere that controls this?
Even more strange, it then logs the final statistic where it says something like
"requestsFinished":119"requestsFinished":119. A number that doesn't make sense at all. Less than the number of actually enqueued links but a lot more than the number of actuall processed pages.Největší nabídka nemovitostí v ČR: 98 210 realit. Nabízíme byty, domy, novostavby, nebytové prostory, pozemky a další reality k prodeji i pronájmu.
188 bytů v aktuální nabídce Byty k prodeji Ústí nad Labem ✓ Parametry hledání: byt, Ústí nad Labem ✓ Největší nabídka nemovitostí v Česku (98 210 inzerátů) s hledáním na mapě a filtrováním s desítkami parametrů ✓
