Vverbal-lime
is there a way to have custom variables accessible inside the crawler function?
Is there a way we can pass in some variables and have them accessible inside the crawler? main use case is to have some internal variables which we can check/modify and execute some conditional logic using them.
for e.g. lets say we have
for e.g. lets say we have
duplicate_countduplicate_count variable (I know we already have this functionality but this is just an example), and I'll update it if the data is already there in my db and stop the crawler if the count exceeds some threshold. Is there a way I could implement this?
