How to solve example.com needs to review the security of your connection before proceeding?

I am trying to scrape a site that has a page that check if the connection is secure and I get this error:

WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Request blocked - received 403 status code.

hurt-tomatoOP•10/6/22, 3:07 PM

I think it needs to wait longer, instead of to shut down after 1 second and to try again, because it always get the error, how can this be done?

Lukas Krivka•10/6/22, 3:21 PM

This is a Cloudflare check for a real browser usually. Try to wait longer, use headless: false and Firefox

hurt-tomatoOP•10/9/22, 8:27 AM

Is there a guide on how to use Firefox?

rubber-blue•10/9/22, 2:37 PM

@pelsec check out this example: https://crawlee.dev/docs/examples/playwright-crawler-firefox

hurt-tomatoOP•10/9/22, 2:59 PM

Thank you