3 replies

why is crawlee running old deleted code for the first url?

main.ts

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

I've updated route.ts and after

npm start

npm start

everything works fine for the last 10 links but for the first link it is running old deleted code for some reason(confirmed by testing multiple times, editing different things). I tried deleting request queue but same thing happened. I'm at a loss for what is happening.

why is crawlee running old deleted code for the first url?

Similar Threads