We are automating Data Warehouse ETL operations which means we have to have a tight execution window and we want to run as many jobs in parallel as possible, but obviously some jobs can't run until others do and that kind of thing is nested and nested at multiple levels.
Problem Statement Example:
1. Job Starts --> Asynchronously Kicks off Process 1, Process 2, Process 3
2. Process 4,5,6 Must wait for Process 2 and 3 to both complete before launching.
3. Process 7,8,9 Must wait for process 1 and 3 to both complete before launching.
4. Process 10,11,12 must wait for process 4, 8 to both complete before launching.
so on and so on...
What We Have Currently:
*** We have a simple single tree like process db table data structure that houses all of these relationship between the processes as well as the process commands themselves, but we are trying figure out how to sit V-Cron on top of this to manage the execution... Control-M does this exact type of thing natively with the UI but of course we couldn't afford it 🙂
*** We really want to make use of our data driven infra and to have vcron just be the engine that reads the data driven events and does the magic of steps 1,2,3,4 above.
Thank you,
Tim Wiley