Hi Support

I am having problems with the newly release 6.07 when dropping large amounts of files into a trigger folder.

I'm running Windows 7 64 bit and I basically dropped 1,500 XML files in to a trigger folder. Everything was running fine and approximately 300 files were processed then everything ground to a halt. No matter what I did I could not re-start the Job, but when I edited it, I got a Server Message (see attached screen shot).

At the time of the error, the VC Server Log reported the following:

09/02/2012 17:57:34 Debug Unhandled error when saving Variables:System.IO.IOException: The process cannot access the file 'C:\Program Files (x86)\VisualCron\settings\variables.xml' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Xml.XmlTextWriter..ctor(String filename, Encoding encoding)
at VisualCronService.VariablesAPI.Save(String strSpecialPath, List`1 lstIncludeOnly, Boolean bolIncludeAll)


Shortly after this, when running a follow-up job, the Server went into overdrive reporting high CPU and memory usage. Checking the log file when this happened, I got the following messages repeated, causing the log file to fill up to almost 2 Gbyte in size.

09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert GeneralLog in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert TaskProcess in local database: The operating system does not support the Encryption Mode provided.
09/02/2012 18:16:10 Err Could not insert JobProcess in local database: The operating system does not support the Encryption Mode provided.

---

I cancelled the VC service, and restarted everything but the Job Trigger that had previously failed wouldn't restart, even when I reselected it, and there are still some 1,200 files in the Trigger folder waiting to run. The only way I can start the process again is to move the files out of the trigger folder and then move them back in again.

It seems to be some kind of contention on the vc 'variables.xml' file.

Any ideas?

Cheers

Andy

We have fixed the infinite loop of logging (that filed your log) in a version update released today:

http://www.visualcron.co....aspx?g=posts&t=2225 

Install this and we are sure that the logging does not go berserk. It would be interesting to know if you can drop the 1500 files after this update. If you experience similar problem again then please send the full log to support@visualcron.com

---

Hi Henrik

Are these two separate errors and are they both resolved? The first one is the contention/IO exception on the variables.xml file, and the second issue was the Encryption Mode message logging.

I just want to prevent these errors occuring at all. I have added a 3 second wait task to the job now after managing my variables just in case the process was running too fast.

Cheers

Andy

I am not sure. But I am sure that the infinite loop worsen the situation. It will be more clear when you try to drop again.

---

Hi Henrik

I was able to do some more testing on this issue today and it still exists. This time there are no errors however, after dropping around 1000 files in to the trigger folder, 93 were processed and then everything just stopped. If I edit the job and select the trigger everything is selected, but when I click okay I get the same server message dialogue I sent previously (Too many changes at once in directory ........).

I have checked the Server Log and at the time everything had stopped there were no errors, although it did say "Saving Jobs" as the last 'Debug' entry:

13/02/2012 13:35:22 Info Task completed: Concatenate Log (2254)
13/02/2012 13:35:22 Info Task started: Delete Log File (2255)
13/02/2012 13:35:22 Info Task completed: Delete Log File (2255)
13/02/2012 13:35:22 Info Task started: wait 2 secs-15 (2256)
13/02/2012 13:35:24 Info Task completed: wait 2 secs-15 (2256)
13/02/2012 13:35:24 Info Job completed: Run PDF Standalone
13/02/2012 13:36:01 Debug Saving Jobs
13/02/2012 15:03:08 Info Job updated
13/02/2012 15:03:08 Debug Saving Jobs

=================================

That was the end of the log file. There are no other VC system errors in the log file.

With the Trigger error message, this was also written to the end of the logfile:
13/02/2012 15:03:08 Info User "VisualCron Default Admin" - Updated Job: Run DocExec Standalone
13/02/2012 15:03:08 Err Trigger error: Too many changes at once in directory:C:\@@Trigger\Philatelic.
13/02/2012 15:03:08 Debug Trigger was expired and inactivated
13/02/2012 15:04:02 Debug Saving Jobs


Cheers

Andy

This is a limit by the system. Currently we use a buffer of 32kB. We can increase it to the maximum of 64kB but that is the maximum by system which means that you cannot drop any more files x.

I am not sure what settings you have but Created + Filename should be enough here. The more you add and the more sub folders you add increase buffer usage.

---

Hi Henrik

Is buffer size an option I can change? If so, where?

Does this mean I can only drop approximately 100 files in to a Trigger folder at one time?

Cheers

Andy

No, you cannot change this. We need to build a new version. We can do this if you want to test.

Number of files for a give buffer varies. I cannot give you an exact number as it varies upon the change, system, architecture etc. I think you need to try this.

But if you do not get an error with 100 today and 101 fails then you will be able to use 200 files in next version - as an example.

---

Hi Henrik

Happy to test this functionality whenever it is ready.

Do you think it could be possible to introduce a feature where we can perhaps list the number of files, e.g. in a folder on an FTP server and if there are more than a set amount (e.g. 75 files) then move the files into a trigger folder in chunks of 75 files?

The only other way I can think of getting around this is to list the files in a folder, and then move them one at a time in a Loop into a 'process' folder, but for 17,000 files that would be a lot of IO, though arguably I guess so would moving 17,000 files :-)

Cheers

Andy

Please try this version:

http://neteject.com/down...Cron/VisualCron6.0.7.exe 

Perhaps the polling Trigger setting could handle that amount?

Have to think about the other thing you are discussing.

---

Hi Henrik

How can I enable the Polling trigger? It is greyed out in the File Trigger and help says it is only used by certain WMI events?

Cheers

Andy

Greyed out? Check the "Use polling". But first, try the new file trigger. We increased the buffer so it would be interesting if more files come through. Polling option is really cpu insensitive, slower and more error prone.

---

Hi Henrik

Dropped 931 files into the trigger folder (2.21Mb in total size) and every single one of them ran through without a problem, so whatever you did seems to have resolved the issue I think

Cheers

Andy

Great, then I guess we hit a limit on the 32Kb buffer that we had. Now using 64Kb (maximum) and I hope this gives enough memory for changes for the future. Otherwise an alternate solution is needed.

---

Yes - the 'List Files' on the FTP task would be a very useful solution too 😉 lol

Thanks again Henrik

Cheers

Andy

I know. We are trying to find open spots in development from this. 🙂

---