Hi All

I just want to confirm something here about the use of JOB variables.

If I set a JOB variable in a job (say {JOB(Active|Id)}_{USERVAR(prefix)}) = {JOB(Active|Id)} and then call another job with the visual cron task, can i pass that variable to subsequent jobs with the original Job ID of the calling job?

(pheew! that was confusing!!!) 😲

Basically I want to create a job specific variable that retain unique values across multiple called jobs for each time a particular job runs

hope it makes sense

cheers

andy

No, original id will be updated to current when using Active.

---

Thanks Henrik - I just wanted to double check.

From a practical point of view, what would be the best way of creating a unique variable name prefix for each job instance that runs so that variable names remain unique across multiple local and remote VC jobs called with the Job/Task Control task? As an example variable name : 'unique_prefix'_FILENAME, 'unique_prefix'_FOLDER, 'unique_prefix'_YYYYMMDD etc but somehow relate that unique prefix back to the originating job that ran?

Basically, we are funelling our daily batch runs through a single job and setting quite a few different variables for each job (such as Date/Time, Folder names, data file names etc) and I don't want subsequent running jobs overwriting variables for current jobs. Then at the end of the workflow, I can just delete the set of variables with the delete variable task, just to keep things tidy.

And I bet the solution is so obvious that I'm missing it!!!

Cheers

Andy

I think the best way is to create a new Variable with Set Variable Task - and use option Translate to constant. Then pass this new Variable.

---

Hi Andy,

did you find a solution to your issue? I have something similar, I have an event file-trigger to trigger an ETL process. The process, task flow, also manages directory/ file copy/ move processes. In the end, depending on the ETL loading process result, it will send an email report.
As I have the ETL process built for parallel loads, I would like to have the jobs, file triggers, work in parallel. What I've observed is that my Job variables are getting overwritten everytime a new file picked up by the trigger.
My expectation is that every job execution would be unique/ isolated and would not share the variables on every trigger. I can 'fix' this by enabling the 'Don't start if a job is already running' but then I won't make use of database parallelization.

Henrik, are there any plans to implement a solution that would help use VisualCron for parallelized solutions like the one I've just described?

Thank you.

Hi Andy,

did you find a solution to your issue? I have something similar, I have an event file-trigger to trigger an ETL process. The process, task flow, also manages directory/ file copy/ move processes. In the end, depending on the ETL loading process result, it will send an email report.
As I have the ETL process built for parallel loads, I would like to have the jobs, file triggers, work in parallel. What I've observed is that my Job variables are getting overwritten everytime a new file picked up by the trigger.
My expectation is that every job execution would be unique/ isolated and would not share the variables on every trigger. I can 'fix' this by enabling the 'Don't start if a job is already running' but then I won't make use of database parallelization.

Henrik, are there any plans to implement a solution that would help use VisualCron for parallelized solutions like the one I've just described?

Thank you.

I am not sure what you mean. Each FIle Trigger call is unique and has its unique Variables if used correctly. Please look at the file trigger tutorial here: https://www.visualcron.com/tutorials.aspx 

---

Hi Henrik,
firstly, I did not mention the trigger per say, I was talking about the Job variables and how on each Trigger response they're reseted. I did look at your tutorials, as suggested, and the first thing said in the tutorial on how to use file triggers is as follows:
"Some variables are temporary, like the file name or file path, it's important to set a special setting for the job"... The next step is the user enabling the 'Put Job in Queue'.
This is exactly what I described in my comment above. This does not allow parallel executions. If we need to put every single trigger in a Job queue, to execute them individually because of 'temporary variables' this means that we can't use parallel execution.
If three files land in the folder at the same time, I need to trigger the job three times, in parallel, and each job must have it's own file_name/ file_path variable to ensure they're handled independently. That's not possible to achieve. Every time a new trigger executes (and it's not put in the queue) it will overwrite the previous file_name and file_path variable, even if I use job variables.

Hi Henrik,
firstly, I did not mention the trigger per say, I was talking about the Job variables and how on each Trigger response they're reseted. I did look at your tutorials, as suggested, and the first thing said in the tutorial on how to use file triggers is as follows:
"Some variables are temporary, like the file name or file path, it's important to set a special setting for the job"... The next step is the user enabling the 'Put Job in Queue'.
This is exactly what I described in my comment above. This does not allow parallel executions. If we need to put every single trigger in a Job queue, to execute them individually because of 'temporary variables' this means that we can't use parallel execution.
If three files land in the folder at the same time, I need to trigger the job three times, in parallel, and each job must have it's own file_name/ file_path variable to ensure they're handled independently. That's not possible to achieve. Every time a new trigger executes (and it's not put in the queue) it will overwrite the previous file_name and file_path variable, even if I use job variables.

I see. Only option is to first run qith queue and capture the value. Then use Job/Task control Task to start external Job but with setting not to wait for completion.

---

Yes, I have done that, but the issue is that each Job call uses the same variables. So the job calls eventually clash and overwrite each other's variables.
For example, I drop two files. It triggers the File Trigger which runs in a queue. Each trigger execution runs a job with the file_name and file_path as Job variables.
The first one runs, and moves the file to a folder and does some work. When that work is done it should move that file to the final folder. The issue is that meanwhile another trigger called the same job, and has overwritten the file_name with a new file_name, so it moves the wrong file to the final folder, leaving the first one in the processing folder.
Really need to have isolated job calls, that while having the same variable_name, are on different memory sections.
If I could only create dynamic variable names, by concatenating a job unique ID to the variable name, but that's not possible as well.
I'm not finding a good way to create parallel execution of a job without overwritting input_variables.
Any ideas?

Yes, I have done that, but the issue is that each Job call uses the same variables. So the job calls eventually clash and overwrite each other's variables.
For example, I drop two files. It triggers the File Trigger which runs in a queue. Each trigger execution runs a job with the file_name and file_path as Job variables.
The first one runs, and moves the file to a folder and does some work. When that work is done it should move that file to the final folder. The issue is that meanwhile another trigger called the same job, and has overwritten the file_name with a new file_name, so it moves the wrong file to the final folder, leaving the first one in the processing folder.
Really need to have isolated job calls, that while having the same variable_name, are on different memory sections.
If I could only create dynamic variable names, by concatenating a job unique ID to the variable name, but that's not possible as well.
I'm not finding a good way to create parallel execution of a job without overwritting input_variables.
Any ideas?

Please create a sample project with as few dependencies as possible to prove this. I assume we are talking about 4 Jobs total And first Job should have 3 Job/Task control Tasks and each Job could have a File write Task that writes the Variable.

---

Not sure I follow, why 3 Job/Task control? Only one, because there's only one target Job that should be run everytime a new file is loaded into the folder. It's the same job, run for each one of the files. That job has to run three times at the same time.
My question is simple; Are the variables of each job execution independent or are they shared (like the trigger File Name)?
My test setup says it's shared, because the second execution, overwrites the Job Variable set by the first execution. So I want to know, is there any way that job execution are independent, don't share the same variable instances.

Just for future notice, I was not able to find a solution for this using VisualCron. Even creating user variables with variable + unique_job_id was did not work. Unique_job_id is the same on two parallel jobs. VisualCron does not support parallel runs of the same Job, variables are always shared by both job instances.
Ended up using VisualCron to trigger database procedures that run in parallel and when they finished wrote to a table. Afterward created a database poller that queried the database and retrieved the entries and completed the processing in VisualCron. Made sure the poller was only enabled when there were entries in the database. When no values were available, the poller would disable.