is there a way to have custom variables accessible inside the crawler function?
Is there a way we can pass in some variables and have them accessible inside the crawler? main use case is to have some internal variables which we can check/modify and execute some conditional logic using them.
for e.g. lets say we have
for e.g. lets say we have
duplicate_count
