Why does crawlee generate warning when both blockedStatusCodes and retryOnBlocked set?
Crawlee generates the following warning when we set both blockedStatusCodes and retryOnBlocked
'Both blockedStatusCodes and retryOnBlocked are set. Please note that the retryOnBlocked feature might not work as expected'
When I set retryOnBlocked to true, crawlee automatically sets default blocked status codes for the session pool an empty array.
When I do handleCloudflareChallenge inside Post navigation hooks, I get 403 status code. Even though handleCloudflareChallenge tries to remove 403 from sessionpool's blockedStatusCodes since it is empty it doesn't have any effect.
Since retryOnBlocked is true, crawlee checks the status code and if its length is zero, it compares with default status codes to check whether to throw an error. And it always throws error.
Is this expected behaviour? If so can I override default blockedStatusCodes? Is there a problem with it?
'Both blockedStatusCodes and retryOnBlocked are set. Please note that the retryOnBlocked feature might not work as expected'
When I set retryOnBlocked to true, crawlee automatically sets default blocked status codes for the session pool an empty array.
When I do handleCloudflareChallenge inside Post navigation hooks, I get 403 status code. Even though handleCloudflareChallenge tries to remove 403 from sessionpool's blockedStatusCodes since it is empty it doesn't have any effect.
Since retryOnBlocked is true, crawlee checks the status code and if its length is zero, it compares with default status codes to check whether to throw an error. And it always throws error.
Is this expected behaviour? If so can I override default blockedStatusCodes? Is there a problem with it?
