other-emerald•2y ago
I was exploring crawlee and trying to undertands some of its termilogies
So currently I have a scenario where in the a tag has a below structure
here you can see that href do not actually have a link indeed its a request back to the server and then server redirects us to the correct url.
Hence enqueue links wont work here, so upon investigation further i came across a playwright util class having a function call
Question: 1. Can the above function be used to click and then get the url of the new page? If not then what would be the right approach of doing so? 2. Secondly I read the documentation and guide in crawlee and i have below doubts, 2.1 Is their a possibility to check which all links have got enqueued by the enqueuelinks function? 2.2 Is there a away I can click on the particular element and get the data from the new page then again come back the main listing page click another url from the list get the data from the new page and the process continues till we hit the last element of the list? 2.3 Can I interrupt the process (Loop)of enqueulinks function in between. I personally find enqueue links function a bit confusing as i am unable to understand the process flow with it so is there any document explaining how request handles and enqueue links works.. I am willing to create a youtube video on once I understand this concepts complete so if anyone interested I can join...
enqueueLinksByClickingElements function i tried by hard luck maybe I have not understood how enqueue links because of which I am unable to use function in a correct manner.Question: 1. Can the above function be used to click and then get the url of the new page? If not then what would be the right approach of doing so? 2. Secondly I read the documentation and guide in crawlee and i have below doubts, 2.1 Is their a possibility to check which all links have got enqueued by the enqueuelinks function? 2.2 Is there a away I can click on the particular element and get the data from the new page then again come back the main listing page click another url from the list get the data from the new page and the process continues till we hit the last element of the list? 2.3 Can I interrupt the process (Loop)of enqueulinks function in between. I personally find enqueue links function a bit confusing as i am unable to understand the process flow with it so is there any document explaining how request handles and enqueue links works.. I am willing to create a youtube video on once I understand this concepts complete so if anyone interested I can join...
1 Reply
flat-fuchsia•2y ago
Please review the documentation below which should have the answers you are looking for and let me know if you have any further questions:
https://crawlee.dev/api/3.5/playwright-crawler/namespace/playwrightClickElements
https://crawlee.dev/api/3.5/playwright-crawler/namespace/playwrightClickElements#enqueueLinksByClickingElements
https://crawlee.dev/api/3.5/core/interface/EnqueueLinksOptions
https://crawlee.dev/api/3.5/core/interface/EnqueueLinksOptions#transformRequestFunction