how to handle different workflows with one or multiple crawlers?
Hello,
I have an express server standing in front of the crawler accepting different types of mutually exclusive workflow requests like a strategy pattern. each workflow works fine if only one strategy runs at a time but none of them can be work concurrently.
I tried 2 separate methods
1. Separate crawler instances but then I get an invalid queue id error
2. Same crawler but use control flow: this results in the requestHandler logic being skipped over and data not being found in the KeyValueStore
Any suggestions for how to handle different workflows concurrently?
I have an express server standing in front of the crawler accepting different types of mutually exclusive workflow requests like a strategy pattern. each workflow works fine if only one strategy runs at a time but none of them can be work concurrently.
I tried 2 separate methods
1. Separate crawler instances but then I get an invalid queue id error
2. Same crawler but use control flow: this results in the requestHandler logic being skipped over and data not being found in the KeyValueStore
Any suggestions for how to handle different workflows concurrently?
