How does one go about using the File Trigger to monitor a folder (where files can be uploaded via ftp) and run the trigger when a file has completed the upload process?

I understand how to create the event, set the credentials, set the folder and file mask and turn on the 'Trigger when file has been released'. However, I am not sure what 'File/Folder watch type' and 'Notify filters' to use.

I have tried 'Created' with 'File Name' which ran the task as soon as the file began uploading.

I have tried 'Created' with 'Size' and nothing happened.

I have tried 'Created' with 'Last Write' and nothing happened.

I have tried 'Modified' with 'Last Write' which ran the task when the file completed its upload (which I thought was good). However, when I tested by uploading 3 files, the job ran 6 times. The job runs a powershell script which will download the file to the server, remove the file from the ftp server and then process the file.

I have the 'Do not run job if it's running' checked and the sub option 'Put Job in queue' is also checked. I would assume that this would place the job into a queue for each file that was uploaded. If that's true then I would have expected 3 jobs to run (for 3 files uploaded), instead of the 6 times that it ran.

You should use default settings. Created + File name. Change may trigger many changes. Also, use put Job in queue option. This way, it will process one file at a time.

---

That was one of the settings that I tried. The problem is that the trigger releases when the file starts to upload. It does not wait until the upload is complete. I have also checked the 'Trigger when file has been released' checkbox, but it still releases before the upload completes.

I am using version 5.4.7.

How large is the file and how long does it take before the upload is complete? Upload speed?

---

We are testing with 3 sizes (8kb, 5mb and 40mb). I am currently testing with the 5mb file. My upload speed is about 110kb/s. It takes about 30 seconds to upload.

I think the problem might be that your FTP server application does not lock the file when it is being uploaded. Normally, when a file is copied from A to B within the system the file is locked during the whole time.

I don't know what is the best approach here but perhaps you should just add a Wait Task as the first Task.

I don't know how we best now that a file is complete - if it is not exclusively locked by the server software.

---

I wondered about that yesterday. I checked our ftp server (TitanFTP) and it contains an option called 'Exclusively lock file during upload' which is turned on. I'll check with the vendor to see if there is an issue with that feature and get back to you.

Have not heard back from the vendor as yet. I plan on upgrading both sets of software to their latest versions this weekend. However, I did create a simple c# program to check if a file is locked by another process. I used this application to test the file being uploaded. It reported that the file was locked (during upload) and then available when the upload completed. I ran this test on the FTP server itself and then through a network share from the vc server.

Can you tell me how the system is checking the file for a lock status? I could then update my test application and try it again.

We try a simple: File.Open(path, FileMode.Open)

But perhaps it works better with the third "access" parameter - please try that on your system.

---

I assume that 'path' contains the path and filename that is trying to be opened. I changed my code to use File.Open. I get the same results. When testing against a file being uploaded it tell me the file is in use by another process (which is correct). When the file completes its upload and I test it again, it tells me that nothing has a lock on the file.

What are my next steps for troubleshooting this?

I think you should try to add a Copy file Task right after Trigger. If the file can be copied they are not locking correctly. Please test that.

---

Ok, first let me say that I have now updated both servers (ftp and visual cron) to their latest versions.

I added a new 'copy files' task to my test job and deactivated my other task. I then uploaded the 5mb file and the task tried to run (and failed). Looking at the job log, the 'copy files' message states: Exception in Task: The process cannot access the file 'xxxx' because it is being used by another process.

That is the same error I get when I use my testing application. This leads me to believe either (1) my file trigger is not setup correctly or (2) the file trigger is not working correctly.

Please let me know what to test next.

Not sure if the problem is that the FTP server is not "keeping" the lock and releases the file from time to time.

What if you use VisualCron and copy something yourself to that path (not using the FTP server). Is VisualCron then able to copy the file?

---

I just used windows explorer to copy a 5 mg file from my local machine to the server. The job started as soon as the copy began. When it was done I checked the job log and saw the same error.

Any other ideas?

We have tested this with many files but cannot reproduce this.

1. Which OS are you using?
2. Could you export your settings and send to support@visualcron.com

---

1) Our servers are Windows 2003 SP2 and workstations are Windows XP SP3
2) Sent via email.

The problem might be that you having a file filter on the Copy file Task. What could happen here is that you try to pick up another *.txt file (that really is in use). Try using Variables instead. See attached image. We tried these settings with success against a remote drive.

Another problem, which I cannot verify since you did not send Credential is if a Credential is used on the Copy files Task.

---

I think we might be going a little afield here. The 'copy file' task in the test job was added into the job by an above request/post. All of our jobs run a single task which is a bat file that runs a powershell script file. All work is done within the script.

Our understanding of the File Trigger is that it can be used to monitor a folder for certain files. Based on the various settings it can then do something. We want the job to trigger when a file is finished uploading. The problem is that the File Trigger is allowing the tasks to run before the file is finished uploading.

I have written a test application (using the File.Open method) to see if the file was not being locked by the ftp server (it was as the application gave the correct error). I added in a 'file copy' task to also test if the file was locked (or not) when the task executed. Again the file was locked and the task failed to run. In all of our testing we are only working with a single 5 mg text file.

I created another file trigger to monitor a local folder/file and then uploaded the same 5 mg file to that folder (using windows explorer). The trigger waited until the file completed upload before releasing (which allowed the task to function correctly).

I have also gone back to the remote file trigger and changed the file name to be the exact file (file_5mb.txt) that we are uploading for tests. The trigger still released as soon as the file began to upload.

I also created an open share on the remote machine (to take the credentials out of the picture) and re-tested the process. The trigger still releases when the upload begins.

It appears that the issue is with using the File Trigger event on a remove drive.

If the remote drive is a normal Windows drive it should not matter. I noticed the name of the server which lead me to that it could be VMWare or similar? Anything that is special about that share?


The "wait for file release" functionality is not any rocket science from our side. Just the code we sent to you.

As I mentioned we tested this on a remote drive as well but it would be interesting if you tested this on another remote drive.

---

Both machines are VMs.

I just created a file trigger to our old ftp server (which is not a vm) and it did the same thing (released before file was uploaded). While the vm should not be causing an issue, I will test it on a non-vm server.

I have a test server here (outside our networked environment) that I will install the trial onto to test. If the trigger works on a non-vm machine, but not a vm machine what will our next steps be?

I am not sure. We have tested this on several machines now. VisualCron does not take any sides but listen to events from the operative system. We have not seen or heard of this issue before and we have a lot of users using the File Trigger.

Let us think about it. It would be great if we had your environment to see what really happens.

---

I understand about the environment, but the best I could do would be to run a version that would write a debug log file that I could send to you.

I downloaded 5.4.9 and installed it on my windows 2003 sp 2 server. The .NET 2.0 SP1 is also loaded. I created a single job that uses a file trigger event and a file copy task. I had to change the file to a 830 mb file so I would have time to switch machines and watch the process (it takes about 60 seconds to upload).

I also have a windows 2003 sp2 server which runs my Filezilla (0.9.33) ftp server. I created a standard shared folder for the Visual Cron server to use.

The job worked, but I am not sure why. I say this because the copy file task was actually executing while the file was still being uploaded to the ftp server. I know this because I was watching the final folder destination and the file appeared. When that happened I switched over to the machine that was performing the upload (windows xp sp3 using Filezilla client 3.3.1) and saw that the file was still uploading.

I used my small testor application to try to access the file on the remote server and it gave the answer of 'Can not access the file xxx because it is currently being used by another process'.

I have copied out the part from the server log (in case it helps).

1/11/2010 1:13:22 PM Debug File trigger event (objectchanged): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (enter lock): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (before counter check): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (in counter check): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (result object created): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (after file release): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (in fire trigger):
1/11/2010 1:13:22 PM Info Job started: File Trigger Test
1/11/2010 1:13:22 PM Debug Job (6) was added to processlist: File Trigger Test
1/11/2010 1:13:22 PM Debug File trigger event (before sending result): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (after sending result): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug File trigger event (exit lock): 1, path: \\192.168.155.10\Fragomen\test\file_830mb.txt
1/11/2010 1:13:22 PM Debug Task (6) was added to processlist: copy file
1/11/2010 1:13:22 PM Info Task started: copy file (6)
1/11/2010 1:14:38 PM Info Task completed: copy file (6)
1/11/2010 1:14:38 PM Debug Setting previous task in TaskProcessCompleted: copy file (58f6e61f-521a-4bdb-a16a-95498aaac224) in job: File Trigger Test
1/11/2010 1:14:38 PM Debug Process status - About to SendTaskProcess (6)
1/11/2010 1:14:38 PM Debug Process status - About to RemoveTaskProcess (6)
1/11/2010 1:14:38 PM Debug Task (6) was removed from processlist: copy file
1/11/2010 1:14:38 PM Debug Job (6) was removed from processlist: File Trigger Test
1/11/2010 1:14:38 PM Debug Next execution (2) for job 'File Trigger Test' is: 12:00:00 AM
1/11/2010 1:14:38 PM Info Job completed: File Trigger Test

I have now got my local environment (which is not vms) to closely mimic our production environment. My file trigger job now executes a batch file which in turn runs a powershell script. It behaves just as production does. The trigger allows the task to run before the file is uploaded.

Here is the server log for the transaction.

1/11/2010 3:55:57 PM Debug File watcher started. Directory: \\192.168.155.10\Fragomen\vcron\in, filter: *.txt
1/11/2010 3:56:09 PM Debug File trigger event (objectchanged): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (enter lock): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (before counter check): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (in counter check): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (result object created): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (after file release): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (in fire trigger):
1/11/2010 3:56:09 PM Info Job started: File Trigger Test
1/11/2010 3:56:09 PM Debug Job (13) was added to processlist: File Trigger Test
1/11/2010 3:56:09 PM Debug File trigger event (before sending result): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (after sending result): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Debug File trigger event (exit lock): 1, path: \\192.168.155.10\Fragomen\vcron\in\file_40mb.txt
1/11/2010 3:56:09 PM Info Task: copy file was skipped because it was inactive.
1/11/2010 3:56:09 PM Info Task: copy file (via batch) was skipped because it was inactive.
1/11/2010 3:56:09 PM Debug Task (13) was added to processlist: fragomen.vc.trigger.test.ps1
1/11/2010 3:56:09 PM Info Task started: fragomen.vc.trigger.test.ps1 (13)
1/11/2010 3:56:09 PM Debug Process status - retrieving process id (13)
1/11/2010 3:56:10 PM Debug Process status - has left WaitForExit (13)
1/11/2010 3:56:10 PM Debug Process status - OutPut captured (13)
1/11/2010 3:56:10 PM Debug Process status - trying to retrieve exit code
1/11/2010 3:56:10 PM Debug Process status - ExitCode fetched (13)
1/11/2010 3:56:10 PM Debug Process status - ProcessHandles closed (13)
1/11/2010 3:56:10 PM Debug Ending timeout timer (13)
1/11/2010 3:56:10 PM Debug Ending output readers (13)
1/11/2010 3:56:10 PM Debug Closing desktop handles (13)
1/11/2010 3:56:10 PM Debug Process status - About to raise TaskCompleted (13)
1/11/2010 3:56:10 PM Info Task completed: fragomen.vc.trigger.test.ps1 (13)
1/11/2010 3:56:10 PM Debug Setting previous task in TaskProcessCompleted: fragomen.vc.trigger.test.ps1 (f83ce5db-5d44-4efc-b989-c02fb01b3da3) in job: File Trigger Test
1/11/2010 3:56:10 PM Debug Process status - About to SendTaskProcess (13)
1/11/2010 3:56:10 PM Debug Process status - About to RemoveTaskProcess (13)
1/11/2010 3:56:10 PM Debug Task (13) was removed from processlist: fragomen.vc.trigger.test.ps1
1/11/2010 3:56:10 PM Debug Job (13) was removed from processlist: File Trigger Test
1/11/2010 3:56:10 PM Debug Next execution (2) for job 'File Trigger Test' is: 12:00:00 AM
1/11/2010 3:56:10 PM Info Job completed: File Trigger Test

I created a new test job on the production machine. The trigger was defined as before (file trigger). I created a single task (file copy) that would copy the last triggered file (based on variables) to the local machine.

It did not work. Still got the same error (file was in use).

We think we have found the problem now. The main reason is that Windows 2003 throws another error than other operating systems. An error that we ignored. We will make some changes and publish a new version here for test.

---