I needed a way to stage data files for continuous import by a VisualCron job. The way I did this was to create 7 folders and stage data within each folder, then when the job would finish importing the last file within folder1 I had 5 tasks that would delete from folder1, move files from folder2 to folder1 and so on. I then wrote out a log file to retrigger the job to restart so it would start importing the next set of data. It works just fine, however is this the most efficient way of doing this?

From your description I am not 100% what you are trying to do. If you are talking about handling backups of VC settings or? Why 7 folders - for the days? Maybe you could name files with dates in it instead. Again, I am not sure what you are doing.

---

I am importing historical data for each week for a client into a new system. So I have a data file created for each week. The import tasks take about 8 hours so to get some sleep on the weekends 🙂 I stage 7 weeks of data into folders. When the import task finishes I have the system move the data into the input directory for the import task then move all the other data up one folder so its staged at the end of the next import. So my VisualCron job looks like this:

Execute [Import data for week 03042013]
Delete files from staging folder IMPORT1
Copy files from staging folder IMPORT2 to IMPORT1
Copy files from staging folder IMPORT3 to IMPORT2
Copy files from staging folder IMPORT4 to IMPORT3
..
Copy log file from 03042013 import to trigger folder
Restart job