rare-sapphire•6mo ago

TypeError: Invalid URL

Adding requests with crawler.run(["https://website.com/1234"]); works locally while in the apify cloud it breaks with the following error: Reclaiming failed request back to the list or queue. TypeError: Invalid URL It appears that while running in the cloud, the URL is split by character and each creates a request in the queue, as it can be seen in the screenshot. The bug happens no matter the URL is hardcoded in the code or added dynamically via input. I'm using crawlee 3.13.0. Complete error stack:

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. TypeError: Invalid URL
2025-03-11T19:21:27.987Z     at new URL (node:internal/url:806:29)
2025-03-11T19:21:27.988Z     at getCookieContext (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:75:20)
2025-03-11T19:21:27.989Z     at CookieJar.getCookies (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:452:23)
2025-03-11T19:21:27.989Z     at CookieJar.callSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:168:16)
2025-03-11T19:21:27.990Z     at CookieJar.getCookiesSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:575:21)
2025-03-11T19:21:27.991Z     at Session.getCookies (/home/myuser/node_modules/@crawlee/core/session_pool/session.js:264:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._applyCookies (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:344:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._handleNavigation (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:329:20)
2025-03-11T19:21:27.993Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:260:13)
2025-03-11T19:21:27.994Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/playwright/internals/playwright-crawler.js:114:9) {"id":"PznVw0jlt50G6EL","url":"D","retryCount":1}

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. TypeError: Invalid URL
2025-03-11T19:21:27.987Z     at new URL (node:internal/url:806:29)
2025-03-11T19:21:27.988Z     at getCookieContext (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:75:20)
2025-03-11T19:21:27.989Z     at CookieJar.getCookies (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:452:23)
2025-03-11T19:21:27.989Z     at CookieJar.callSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:168:16)
2025-03-11T19:21:27.990Z     at CookieJar.getCookiesSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:575:21)
2025-03-11T19:21:27.991Z     at Session.getCookies (/home/myuser/node_modules/@crawlee/core/session_pool/session.js:264:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._applyCookies (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:344:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._handleNavigation (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:329:20)
2025-03-11T19:21:27.993Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:260:13)
2025-03-11T19:21:27.994Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/playwright/internals/playwright-crawler.js:114:9) {"id":"PznVw0jlt50G6EL","url":"D","retryCount":1}

4 Replies

Hall•6mo ago

Someone will reply to you shortly. In the meantime, this might help:

gtry•6mo ago

Hi, could please share some code?

Alexey Udovydchenko•6mo ago

Please try template https://console.apify.com/actors/templates/js-crawlee-puppeteer-chrome - if direct URL works for the template, then you doing something wrong

Apify

Apify Console

Manage Apify, a full-stack web scraping and data extraction platform.

Oleg V.•6mo ago

Also, please share the exact URL you're trying to process. A link to your run on the platform would be helpful as well.

TypeError: Invalid URL

Did you find this page helpful?