Pass args to handler
Hey
I have a crawler which scrapes a lot of different websites, each with multiple urls.
Each website has an associated id, I need for the dataset.
So I want to scrape the urls, get the data but then instantly send it to a database, so I don't have to keep it on the EC2 instance.
Is there a way to pass extra variables to @router.default_handler
for company in valid_company_urls:
crawler = await create_crawler(config, company)
# Run crawler for this company's URLs
await crawler.run(company['url'])
So when I do something like this. How could I pass additional arguments to run that are then passed to the handler.
I have not found anything in the docs.
Thanks for any hints!
I have a crawler which scrapes a lot of different websites, each with multiple urls.
Each website has an associated id, I need for the dataset.
So I want to scrape the urls, get the data but then instantly send it to a database, so I don't have to keep it on the EC2 instance.
Is there a way to pass extra variables to @router.default_handler
for company in valid_company_urls:
crawler = await create_crawler(config, company)
# Run crawler for this company's URLs
await crawler.run(company['url'])
So when I do something like this. How could I pass additional arguments to run that are then passed to the handler.
I have not found anything in the docs.
Thanks for any hints!
