Hi Erik

I need to create 4 different Profile files (a bit like .ini files with name/value pairs) to trigger off external processes for document composition and printing software, so there is a requirement for quite a bit of value matching for replacement variables within each profile. The incoming data file name is the trigger for all of this.

Traditionally this is done with Perl scripts but they have become so difficult to maintain the remit is to minimise the dependency on them as much as possible - so no external tools allowed if at all feasible.

That said (and I think Henrik will probably be horrified with this!) I am actually trying out a call from VC to the Automise 4 tool, which does have looping and mapping capabilities amongst some other nice enterprise features, so, while not ideal, it does work very well and it just might be a better solution than the very expensive alternative!

I hope Henrik can pull a magic rabbit out of his hat on this one :-)

Cheers

Andy

I am actually trying out a call from VC to the Automise 4 tool

Here in Sweden we have a saying translated to "Do not curse in the church" 😁

We'll see what we can do. We appreciate all feedback at this stage.

---

Here's my looping constructoin in VC:

TaskNr - description (tools)

---

1. With a DIR command I get all the dirs from one location and I strip all the non info out of there and I save this in a file. (DIR, LMOD)
   
2. Disable task 1
   
3. Get the 1 line from the file from task 1 which is not starting with :DONE: (FINDSTR, LMOD)
   
4. If task 3 has output continue, else skip. Look in csv file for output task 3 and get data from next column (FINDSTR, LMOD)
   
5. If task 3 has output continue, else skip. DO SOMETHING WITH DATA TASK 4.
   
6. If task 3 has output continue, else skip. Edit the dir list from task 1. Look for the output from task 3 and place :DONE: in front of it. (FIND, LMOD)
   
7. If task 3 has output continue, else skip. ReRun this complete job.
   
8. Enable task 1.
   

My job runs several times, but only the 1st time my dirlist is created. If all the dirs are processed, the loop stops, task 1 is enabled again. Than I can start my job again and the complete looping process is starting over.
I use LMOD a lot, this is a handy tool i use for text manupulation.

I hope I enspire you using VC and no other automating tools which the name I shall not use 😊.

Regards,
Erik

---

Thank you Erik

I will look into Lmod. And in future I won't mention the other 'A' word ;-)

Cheers

Andy

Ok, so we have given it some more thoughts before starting the development. We decided that we first want to add looping support on the whole Job. There are several reasons for this; one is that this is less complex and the second is that we have not decided yet how to both visually and "programatically" design the loop at Task level. We want to see that everything works before messing up things in the Task loop.

The idea and interface for Job looping is basically the same as before. See image.

We also attached an image with lousy handwriting 😁 which explains the flow. When a Job is being run Conditions are checked first, then if Looping is checked all Tasks will be looped according to loop settings. During the loop there will be something called Loop Context Variables. This are normal Variables but context sensitive for the Tasks in the actual loop.

This could be:

- the current value
- number of values in the loop
- what type of loop (for,foreach or while)
- start value
- end value

We would like to get some final feedback on this idea.

One basic question is if Conditions should be evaluated before the loop starts or if loop should be started *before* and include Job Condition check each time. This is a design and performance question.

Another question is if you request any more information in the Loop Context Variables.

---

Hi Henrik

Okay, from my viewpoint and more specifically about what I wish to achieve personally with Looping functionality:

1. I am thinking that LOOP takes precedence over CONDITION, for the simple reason that a Condition could be used to Drop out of the loop? Though that said, I think it may mean a new construct would need to be added to the Conditions 'On Match' Actions to Set Variable Value (i.e. the Counter variable) as well as the RUN Task/Job? This means a Loop could be prematurely ended and therefore more performant and another job/task run automatically?

2. Will the start/end values be addressable (i.e. as an Array) in a further loop? If so I think Context variables will be fine as long as they are usable in an easy manner.

3. Functionaly, Loops at the Task level would be a far better option for me, but nothing to stop us having one job calling a loop job and then back again?

Just my 2 pence worth :-)

Cheers

Andy

1. I am thinking that LOOP takes precedence over CONDITION, for the simple reason that a Condition could be used to Drop out of the loop? Though that said, I think it may mean a new construct would need to be added to the Conditions 'On Match' Actions to Set Variable Value (i.e. the Counter variable) as well as the RUN Task/Job? This means a Loop could be prematurely ended and therefore more performant and another job/task run automatically?

Ok, sound reasonable.

2. Will the start/end values be addressable (i.e. as an Array) in a further loop? If so I think Context variables will be fine as long as they are usable in an easy manner.

Yes, it should be easy to access those.

3. Functionaly, Loops at the Task level would be a far better option for me, but nothing to stop us having one job calling a loop job and then back again?

Yes, maybe it would. But as I mentioned in the last post we go for Job first because it is easier to implement and let us test the basics before adding it to the Task level.

---

We have done some further thinking about looping. We are thinking about skipping the Job looping in first phase and go directly to Task looping.

One of our problems when dealing with this is how to present loops in the interface. Both configuring loops and listing Task items with loop.

We have tried, in the example images below to keep it very simple. Here are the basic ideas:

• Loops are added from Add/Edit Job->Task listing
  
• Loops can be added by clicking on Add loop button or right click on a Task
  
• Loops are configured in the Add Loop window
  
• You can choose a start Task for loop and an end task in loop.
  
• You can configure the color of the "loop connector" - the part that shows that one or more Tasks belong to a loop.
  
• In the Loop settings of Add loop window you configure the specifics for the loop.
  
• You can have as many loops as you want. But not more than the number of Tasks and
• Loops cannot cross each other.
  
• Loop is being made in Task order

The idea with the "loop connector" is that you can see which Tasks belong to which loop (color). The disadvantage with using the loop connector with start and end connectors is that it will look strange when Tasks are sorted in a different way than order. On the other hand it might be easier to understand when sorting is right. We have not decided if we are going to show this in main grid window (for performance) but the idea is to show this in Task list of Job window.

If we go into specifics of API and objects the Loop part will be a part of the Job - not the Task objects. The JobClass will contain a list of Loops which refer to one or more Tasks.

Please look at the images (which are mockups) and give us any feedback about the suggested solution. Mainly, if it is clear enough, any thoughts about the connectors or comments about limitations in the list.

In one of the samples you can see that, for example, Task 1 and 2 are part of one loop. The green connector to the left is marking that.

---

Hi Henrik

To me this looks far better than the Job level looping. Everything seems logical (although I am a little confused by the 'Connector' screenshot) - is this part of the proposed solution or is it simply to demonstrate how tasks are connected?

For the sorting of columns, perhaps it would be an idea to output the 'Task Order' counter inside the Coloured Loop bar so that even if the columns are sorted differently, you can see the order in which the tasks exist within the loop?

Cheers

Andy

To me this looks far better than the Job level looping. Everything seems logical (although I am a little confused by the 'Connector' screenshot) - is this part of the proposed solution or is it simply to demonstrate how tasks are connected?

The connectors show the "span" of the loop. From Task x to Task y. Currently we have two proposed ways of showing this either a colored line to the left of the connected Tasks. Or a the alternate connector screen shot with "bends" on the connectors to show start/end a little clearer.

If more than one loop exist they will have different colors.

For the sorting of columns, perhaps it would be an idea to output the 'Task Order' counter inside the Coloured Loop bar so that even if the columns are sorted differently, you can see the order in which the tasks exist within the loop?

I am not sure what you mean as Task order column is already there. It could be confusing with 2 different orders. If you are talking about the main window listing (at startup) the order column is possible to add (it is just hidden by default). Column sorting is not saved in this window so each time you open it you get the correct listing. I think it is a matter of just knowing or learning of how it works.

---

Okay - all makes sense now - I think for me the 'Bend' connectors are a little clearer, but others may have a different view

Understand now with the Task numbers - I didn't even think to switch them on in the default Job view so that would also help - now that I have discovered them I prefer them enabled :-)

Cheers

Andy

Hi Devteam,

Nice to have some new replies in this post. 😍

In the past I spoke of two seperate tasks for starting and ending a loop. The current proposal looks like this so I love this solution.
Having this done at job level with colours you have a complete overview of your loopings. I find the coulours now a bit hard to see in the task list though.

Having it this way you could have one task getting all the loop info and the following tasks use this in the loop settings as variable.

Just wondering how to address the looping variables. Will there be a job.loop.[name/nr/colour].<looping values> variables?

The conditions, you can use to drop out of the loop, but you might want to use the set variable task for this with a normal condition. The set variable could set the looping itteration counter to the total of itterations, but an action of jumping out of the current loop could do the trick also, it might be better. But I think changing the loop counter shoud work also. That's like jumping back into time...

With the change order button in the job view, you have to wacht out with the looping. You could place a task in a different loop, or will the loop break? There has to be some thinking here.

There might be an extra option in the task at the on error tab: jumping out of the current loop.

The gridview in the screenshot is not as clear as it should be I think. I think the tasknames should be there also. But they are only mockups, right?

This was my brainwave at the moment, more to come.

Thumbs up! 👍

Regards,
Erik

---

Hi Devteam,
In the past I spoke of two seperate tasks for starting and ending a loop. The current proposal looks like this so I love this solution.

We skipped the "Task type of loop" because it is really not a Task. Also, it would not have made displaying the looping easier. This is more than an ordinary loop and for that reason it just can't be dragn' drop. I hope that you like this solution anyway!

Having this done at job level with colours you have a complete overview of your loopings. I find the coulours now a bit hard to see in the task list though.

Yes, the mockup was bad. I think we will used the "bended" connectors and also indent it a bit out (like in the "bend" example) to make it more clear.

Having it this way you could have one task getting all the loop info and the following tasks use this in the loop settings as variable.

Just wondering how to address the looping variables. Will there be a job.loop.[name/nr/colour].<looping values> variables?

Correct. We are not sure about how the Variables will look like exactly but all values should be easily reachable. We see no need right now to specify a Loop id/name etc. because our Variables will be context based and if you use a Loop Variable in a Task within a loop it will always point to the current loop.

The conditions, you can use to drop out of the loop, but you might want to use the set variable task for this with a normal condition. The set variable could set the looping itteration counter to the total of itterations, but an action of jumping out of the current loop could do the trick also, it might be better. But I think changing the loop counter shoud work also. That's like jumping back into time...

Task Conditions will always be evaluated before the Task is run. I agree that one addition could be to break out the loop in some way would be good.

With the change order button in the job view, you have to wacht out with the looping. You could place a task in a different loop, or will the loop break? There has to be some thinking here.

Yes, it could be messy. We will add warnings when trying to add the Job when order has changed in a wrong way, for example if End Task is before Start Task or a Task has been removed. If a loop covers Task 1 and Task 2 and we move order of Task 2 after Task 3 then the loop will cover all 3 Tasks and we will try to redraw loops.

There might be an extra option in the task at the on error tab: jumping out of the current loop.

Nice addition!

The gridview in the screenshot is not as clear as it should be I think. I think the tasknames should be there also. But they are only mockups, right?

Yes, correct.

This was my brainwave at the moment, more to come.

We like your brainwaves 😁 Thanks!

---

It would also be interesting if you could give us some feedback regarding Tasks and Variables that needs to be updated to support Looping.

For example, we will add a Get Files Task to return a list of files that can be used in loop.

FTP Download/Upload might want need to have a Variable for accessing uploaded/downloaded files, it might already exist, but we need to ensure that format is right so we can just put the Variable in the loop fields.

So, if you have any ideas of common Tasks that you need to extended with new Variables or that you need new Tasks then please let us know!

---

Hi Henrik

Here are some ideas from me based on real customer requirements

= Get multiple files including recursion through directories (Filesystem and FTP)
= Iterate multiple rows and columns in a spreadsheet
= Iterate rows and delimited fields in a txt/csv file
= Append Variables to existing variable list
= String Search/Replace in repeated fields (big one for me) :-)
= PGP Encrypt/Decrypt
= Wait for trigger file (e.g. In a loop, wait for a particular file name to appear before processing all files in a loop). This would help with synchronisation of loop processing of files until they have all appeared.

I'm sure I may well come up with some more :-)

Cheers

Andy

Hi Henrik

Here are some ideas from me based on real customer requirements

= Get multiple files including recursion through directories (Filesystem and FTP)
= Iterate multiple rows and columns in a spreadsheet
= Iterate rows and delimited fields in a txt/csv file
= Append Variables to existing variable list
= String Search/Replace in repeated fields (big one for me) :-)
= PGP Encrypt/Decrypt
= Wait for trigger file (e.g. In a loop, wait for a particular file name to appear before processing all files in a loop). This would help with synchronisation of loop processing of files until they have all appeared.

I'm sure I may well come up with some more :-)

Cheers

Andy

Ok, could you elaborate more on String Search/Replace and put it in a separate Feature request.

---

We are thinking, while we are at it, to add another tab in the Task window "On success". This controls what happens if the Task succeeds. Of course, default value is "Continue with next Task in order".

Any ideas what you would like to see here?

---

Hi,

Does this tab has any relation with looping or appears this tab in every task?

regards
Erik

---

Hi,

Does this tab has any relation with looping or appears this tab in every task?

regards
Erik

The idea is that this appears in every Task and has similar alternatives like "On error". One could argue against this functionality as you already have arranged your Tasks in order. If the need for this functionality is there then we will add this.

---

Hi,
Does this tab has any relation with looping or appears this tab in every task?

The idea is that this appears in every Task and has similar alternatives like "On error". One could argue against this functionality as you already have arranged your Tasks in order. If the need for this functionality is there then we will add this.

I was thinking about this this night and thought why this is needed. Everything you want, you can do it in the next task after a task succeed.
I thought about jumping over tasks, e.x. if a task fails you let it continue to do an error task and then another task stopping the job. If it wass succeesful you jump over these two tasks, but this can be configured right now as well with the use of conditions.
So bottomline, I'm not sure what the bennefit is for this addition. I might overlook something though...

Regards,
Erik

---

Hi Henrik & Erik

I think the OnSuccess Tab could be an extremely useful tab for each task. For example, consider a business process workflow, when a particular task is completed successfully (or in error) it could signal a 'checkpoint' within a workflow and automatically update a database table to flag the step as success or failure. Yes, this can be achieved through conditions and sub tasks but to drive it through a single self documenting tab would be far simpler in my view?

Cheers

Andy

We have a beta now. Please report any bugs, ask any questions + feedback in this thread here:

http://www.visualcron.co....aspx?g=posts&t=2013 

---

Thank you!! The looping feature was sorely needed. One of the things that delayed me from buying VC (but I finally did anyway). I will have to check out the beta. I also vote for the onSuccess tab.