#crawlee-python - Apify & Crawlee

Can your LinkedIn Company Employee Scrapers access 'People' tab

Bo12/9/25, 4:41 AM

without a LinkedIn session cookie (li_at)? I’m on the Free plan and the actor has no field for a cookie. Before I upgrade, I need a yes/no: does this actor support authenticated scraping at all, and if so, where is the cookie input located? If it cannot scrape employee lists without authentication, then upgrading won’t fix anything, because it will still be blocked at the UI level. The results I'm getting are mixed at best and I'm told it's because it is not an authenticated search.

Looking for a new Job.

! Bl@st11/28/25, 4:01 PM

I’m an AI engineer with hands-on experience building advanced, end-to-end AI systems across multiple domains.
My work spans multimodal chat systems, voice, generative ad-video pipelines and intelligent automation tools that combine LLMs, computer vision and speech technologies.
I’ve developed solutions for industries like healthcare, civil engineering, finance and recruitment optimizing workflows and reducing manual effort by up to 90%.
From OCR and NLP-driven document analysis to real-time video and avatar systems, I specialize in creating AI products that are both practical and scalable.

My expertise covers :

Build AI agents & multi-agent systems (GPT-4o, Claude 3, CrewAI, AutoGen)
Automate workflows with Zapier, n8n, Make.com, and custom Python scripts
Develop NLP - powered chatbots and memory-based assistants
Integrate voice & speech systems (TTS/STT, Voiceflow, OpenAI Realtime)
Deploy custom LLMs and fine-tuned AI models
Work on Web3, smart contracts, and AI-integrated blockchain games

I’m open to working with startups, research teams, or bold innovators in AI or automation.

If you’re building something interesting or need help with an AI feature, I’m always down to chat.

I need discord bot and scrapper

Original message was deleted

Failed to make outgoing request to sample ZIP file

za11/11/25, 5:11 PM

My actor tried to call the ZIP file via URL but error. I want to check if the runner can perform an outgoing URL?
URL: https://getsamplefiles.com/download/zip/sample-1.zip

Strange behaviour when using rq_client()

Eric11/3/25, 3:06 PM

I have been struggling with tests for a while, and finally reduced it to a simple test for which I don't understand the behaviour. Is this expected (and I am missing sth) or is this expected?

This test fails
async def test_failing():
storage_client = MemoryStorageClient()
request_queue_client = await storage_client.create_rq_client()

req = Request.from_url("https://crawlee.dev")
await request_queue_client.add_batch_of_requests([req])

crawler = BasicCrawler(
concurrency_settings=ConcurrencySettings(desired_concurrency=1, max_concurrency=1),
max_crawl_depth=2,
storage_client=storage_client,
)

@crawler.router.default_handler
async def handler(context: BasicCrawlingContext) -> None:
pass

stats = await crawler.run()
assert stats.requests_finished > 0

but this one passes

async def test_success():
storage_client = MemoryStorageClient()

req = Request.from_url("https://crawlee.dev")

crawler = BasicCrawler(
concurrency_settings=ConcurrencySettings(desired_concurrency=1, max_concurrency=1),
max_crawl_depth=2,
storage_client=storage_client,
)

@crawler.router.default_handler
async def handler(context: BasicCrawlingContext) -> None:
pass

await crawler.add_requests([req])

stats = await crawler.run()
assert stats.requests_finished > 0

the only difference is that in the first I add requests through a request client or through the crawler. If I add it through this
rq = await RequestQueue.open()
await rq.add_request(req)

it also fails. Thanks in advance

Downloading pdfs and other files

Eric11/3/25, 9:54 AM

When crawling, whenever a download starts in a webpage (pdf or similar), crawlee errors. What would be the correct way to catch this error and do the download myself? I have a similar problem with XMLs. In other words I am using a Playwright crawler but I want to be able to download content (and parse + enqueue links) on my own when my crawler can't. I was thinking on having another request queue for pdfs that I dequeue using another crawler (and add to this queue infering it is a pdf from the url for example), but I was hoping there was an easier way, like defining a fallback parser or similar. Thanks in advance!

i face problem when listen abort event [python sdk]

flash mage10/29/25, 9:15 PM

I followed the official documentation's template code to implement an Apify Actor, built and deployed it on the Apify platform.
When I run the Actor and click Abort (graceful abort), the ABORTING event handler handler_foo does not log anything — Actor.log.info(...) inside it never appears.
I have no way to confirm whether the handler is actually being triggered, and I need to perform custom actions when the user aborts the run.
Questions:

How can I guarantee that the ABORTING event handler is called?
How can I ensure that logs and operations (like Actor.set_value) inside it execute and appear in the logs?

--------------------------------------------------------------------------------------------
from apify import Actor
from apify_shared.consts import ActorEventTypes
import asyncio

def handler_foo(arg: dict):
Actor.log.info(f'handler_foo: arg = {arg}')

def handler_boo(arg: dict):
pass

async def main():
async with Actor:
# Add event handler
Actor.on(ActorEventTypes.ABORTING, handler_foo)
await asyncio.sleep(5*1000)
# Remove all handlers for a specific event
Actor.off('systemInfo')

# Remove a specific event handler
Actor.off('systemInfo', handler_boo)

Field required [type=missing, input_value={'id': 'e65qXc7GXZBNFA8CD...runs/e65qXc7GXZBNFA8CD'}, inpu

Kaito10/29/25, 4:06 PM

https://console.apify.com/view/runs/e65qXc7GXZBNFA8CD

My actor suddenly threw an error, and I didn't make any changes to the code. What's more, it works fine when I run it myself, but there's an issue when my users run it.

Apify

Apify Console

Manage Apify, a full-stack web scraping and data extraction platform.

AdaptativePlaywright crawler with SQL queue gets stuck

Eric10/28/25, 9:21 AM

I am using multiple instances of an AdaptativePlaywright crawler in different servers, using the SQL queue. After running for some time, it looks like the crawler stops working and I only regularly see this logs:
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 9; desired_concurrency = 10; cpu = 0.0; mem = 0.0; event_loop = 0.023; client_info = 0.0 │
│ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │
│ [crawlee.crawlers._basic._basic_crawler] INFO Crawled 0/1157424 pages, 0 failed requests, desired concurrency 10.

Restarting the pod where the crawler runs doesn't fix the issue

Crawl Speed

Drogbata10/28/25, 7:46 AM

Lets say i have one session that makes 15 requests. I do not like that it finishes crawl in 25 seconds. How to stretch crawl duration / slow down speed of crawl for small ones? max_requests_per_minute is useless in this situation right? What options i have?

Get A Job Offer

Original message was deleted

How to maximize throughput with concurrency settings?

Eric10/27/25, 2:35 PM

I am deploying crawlee in my kubernetes cluster and first I tried not setting a maximum (nor a minimum) of concurrent tasks, but crawlee kept on taking more and more memory until the pod was killed by the cluster (took too much memory). Now I set a conservative maximum but I see resources are very underutilised in some moments depending on the page I am crawling. Is there a way I am not seeing on how to do this correctly? Or is it possible that there is a bug when determining a max amount of concurrent tasks where crawlee spawns too many of them?

Whole crawler dies because "failed to lookup address information: Name or service not known"

Eric10/27/25, 8:01 AM

I am not able to reproduce it in a simple example (it may be a transient error), but I have gotten this error regularly and it kills the crawler completely.

Traceback:
  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1366, in __run_task_function
    if not (await self._is_allowed_based_on_robots_txt_file(request.url)):

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1566, in _is_allowed_based_on_robots_txt_file
    robots_txt_file = await self._get_robots_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1589, in _get_robots_txt_file_for_url
    robots_txt_file = await self._find_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1599, in _find_txt_file_for_url
    return await RobotsTxtFile.find(url, self._http_client)

  File "crawlee/_utils/robots.py", line 48, in find
    return await cls.load(str(robots_url), http_client, proxy_info)

  File "crawlee/_utils/robots.py", line 59, in load
    response = await http_client.send_request(url, proxy_info=proxy_info)

  File "crawlee/http_clients/_impit.py", line 167, in send_request
    response = await client.request(

impit.ConnectError: Failed to connect to the server.
Reason: hyper_util::client::legacy::Error(
    Connect,
    ConnectError(
        "dns error",
        Custom {
            kind: Uncategorized,
            error: "failed to lookup address information: Name or service not known",
        },
    ),
)
exited with code 1

Traceback:
  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1366, in __run_task_function
    if not (await self._is_allowed_based_on_robots_txt_file(request.url)):

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1566, in _is_allowed_based_on_robots_txt_file
    robots_txt_file = await self._get_robots_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1589, in _get_robots_txt_file_for_url
    robots_txt_file = await self._find_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1599, in _find_txt_file_for_url
    return await RobotsTxtFile.find(url, self._http_client)

  File "crawlee/_utils/robots.py", line 48, in find
    return await cls.load(str(robots_url), http_client, proxy_info)

  File "crawlee/_utils/robots.py", line 59, in load
    response = await http_client.send_request(url, proxy_info=proxy_info)

  File "crawlee/http_clients/_impit.py", line 167, in send_request
    response = await client.request(

impit.ConnectError: Failed to connect to the server.
Reason: hyper_util::client::legacy::Error(
    Connect,
    ConnectError(
        "dns error",
        Custom {
            kind: Uncategorized,
            error: "failed to lookup address information: Name or service not known",
        },
    ),
)
exited with code 1

This is my crawler:

crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
  playwright_crawler_specific_kwargs={
      "browser_type": "firefox",
      "headless": True,
  },
  max_session_rotations=10,
  retry_on_blocked=True,
  max_request_retries=5,
  keep_alive=True,
  respect_robots_txt_file=True,
)

crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
  playwright_crawler_specific_kwargs={
      "browser_type": "firefox",
      "headless": True,
  },
  max_session_rotations=10,
  retry_on_blocked=True,
  max_request_retries=5,
  keep_alive=True,
  respect_robots_txt_file=True,
)

I am on version 1.0.4 and I was crawling crawlee.dev (though it doesn't fail in a specific page)

For 429 errors, do you consider adding a setting or change the default behaviour to allow retries?

Eric10/20/25, 11:36 AM

From what I have seen, when there is a 429 error (rate limit), Crawlee tries rotating the session, but if all of the sessions it uses return 429 (for example all proxies have been rate-limited) the request is just marked as handled and forgotten. I would expect this to be considered a transient error that can be fixed with time, same as a 500 error. Do you expect to support this change in behaviour in the future?

Not able to connect Kiwi MCP Server through my OpenAI agent workflow

rudra10/13/25, 9:26 AM

You can see the attached screen, I'm trying to connect the MCP server inside my Agnet workflow, but It is giving following error,
{
"error": {
"message": "Error retrieving tool list from MCP server: 'http://wondrous-director--kiwi-mcp-server-task.apify.actor/mcp'. Http status code: 424 (Failed Dependency)",
"type": "external_connector_error",
"param": "tools",
"code": "http_error"
}
}

The (python-client) Group Scraper does not fetch the post title, only the post description.

Omar10/11/25, 3:55 AM

In the given image, using python-clientpython-client I can only get the post description but I cannot fetch the title of the post itself (shown in bold).
Any clue how to solve this?

Session Cookies

Drogbata9/21/25, 9:05 PM

When i use bs4 crawler i would expect that session stores received cookies and then reuses in next requests but SessionCookies are empty. Any idea why and how to make it work?

How to set RAM when calling an Actor via API (Python client)?

emsinhvien9/17/25, 9:21 AM

Hi,
I’m running the novi/fast-tiktok-api Actor from my Python app using the Apify Python Client.

Here is a simplified version of my code:

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run_input = {
"limit": 10,
"region": "JP",
"type": "TREND"
}

run = client.actor("novi/fast-tiktok-api").call(
run_input=run_input
)

I would like to set the RAM (memory) for the Actor run, for example 128 MB or 256 MB.
I tried passing memory=128 directly to .call(), but I got the error:

ActorClient.call() got an unexpected keyword argument 'memory'

Could you please show me the correct way to configure RAM (memory) when calling an Actor from the Python client?

Thanks a lot!

Have a subscription but can't run anything!

Anakin9/11/25, 9:54 AM

Hello, I've used apify before, infact I have an active subscription, but from this morning (don't know why) I can't run any actor and can't set the maximum charged results. I try to change browser, empy the cache but nothing works. This is the error I got: "Error: Cannot run Actor (Expected property maxItemsmaxItems to be of type numbernumber but received type stringstring in object)". Anybody can help me please'

Pay-per-Event testing

MrBrogrammer9/7/25, 5:03 PM

Jo how do I test the monitization with Pay per Event if I run it on apify and not locally on my device?