why is crawlee running old deleted code for the first url?

main.ts

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

I've updated route.ts and after npm startnpm start everything works fine for the last 10 links but for the first link it is running old deleted code for some reason(confirmed by testing multiple times, editing different things). I tried deleting request queue but same thing happened. I'm at a loss for what is happening.

Apify & Crawlee•2y ago•

3 replies

significant-gray

why is crawlee running old deleted code for the first url?

main.ts

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

import { PlaywrightCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = [
  11 links
  ...
];

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    maxRequestsPerCrawl: 1,
    maxRequestRetries: 0,
    maxRequestsPerMinute: 16,
    maxConcurrency: 1,
    headless: false,
});

await crawler.run(startUrls);

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

14,735 Members

Join

why is crawlee running old deleted code for the first url?

why is crawlee running old deleted code for the first url?

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads