Apify Discord Mirror

Updated 2 months ago

Debug and troubleshooting Crawlee

At a glance

The community member is experiencing an issue with Crawlee, where errors in the response-handler are not being properly reported. Instead, Crawlee retries the request 9 more times without showing any error message. The community member provides an example of a syntax error in a query selector, where the closing bracket is missing. They are asking if there is a way to troubleshoot these kinds of issues.

In the comments, another community member tags Vlada Dusek and asks them to check the issue. Additionally, another comment mentions that a future release of Crawlee will include a parameter to terminate the run immediately when an error is received, which may help address the issue.

When there is an error in the response-handler, then it does not show an error, but it will fail unnoticed, and Crawlee will retry 9 more times.

To illustrate the following syntax-error:
element = await context.page.query_selector('div[id="main"')

It should be with a closing bracket after "main":
element = await context.page.query_selector('div[id="main"]')

However, instead of failing, Crawlee will proceed and keep trying to query the page and it will not show any message on where it failed.
Is there any way to troubleshoot these kind of issues?
O
M
2 comments
cc @Vlada Dusek Can You check it please πŸ™
Pay attention to this PR.

When it reaches release, a parameter will be added that will allow the run to terminate immediately when an error is received.
Add a reply
Sign up and join the conversation on Discord