extended-salmon•2y ago
Callback after routing
Hi guys, im a junior developer assigned to use the crawlee library to create a function to validate our data. Basically, they want it so we dont have to edit every scraper we make everytime we want to change this.
Would there be an option I could set in routers or crawlers to be able to
1. Call a function after a certain router
(Really we just want to validate data without editing every single scraper so if we could do this before pushing thatd be preferrable.)
2. Validate data.
3. Return error if data is incorrect
Really I just need help with the first part.
I am still in college and this is my first job working through something independently like this. Any help would be appreciated.
1 Reply
You can have the validation as a code that you import to each project and then you just do it every time before Dataset.pushData