We have had ‘catastrophic error’ occur several times now caused due to FTP tasks and none of the FTP tasks work after this error has occured until we restart the VC service. The good news is we got more logging on it this time than last. This was after rebooting on Wednesday (9/19). Henrik, I know we exchanged some emails on this issue and at one point you mentioned the .NET version on the server. Would you recommend installing .NET4? Have you been able to confirm whether that is the root cause? Since this is only server with heavy FTP activity, we can't reproduce this issue on another server. Even on the same server, we are not sure how to reproduce the issue. Has any other users seen this problem?

9/22/2012 12:42:47 AM Debug Calling StartTaskProcess() with Task: Move all other .zip file to H:/TempZIPFilesToFTPToIAD (0)
9/22/2012 12:42:47 AM Debug Task (1536) was added to processlist: Move all other .zip file to H:/TempZIPFilesToFTPToIAD
9/22/2012 12:42:47 AM Err Could not insert GeneralLog in local database: Internal error: Unable to successfully execute disk IO on the file system.
9/22/2012 12:42:47 AM Debug Entering wait loop. Move all other .zip file to H:/TempZIPFilesToFTPToIAD (1536)
9/22/2012 12:42:47 AM Info Task started: Move all other .zip file to H:/TempZIPFilesToFTPToIAD (1536)


9/22/2012 12:58:00 AM Err Unhandled exception in ProcessTaskFTP.Run: System.ApplicationException: Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))
at System..Threading.ReaderWriterLock.ReleaseWriterLockInternal()
at SBSharedResource.TElSharedResource.Done()
at SBCryptoProvBuiltIn.TElBuiltInCryptoProvider.InternalCreateKey(Int32 Algorithm, Int32 Mode, Byte[] AlgOID, Byte[] AlgParams, TElCPParameters Params)
at SBCryptoProvBuiltIn.TElBuiltInCryptoProvider.CreateKey(Int32 Algorithm, Int32 Mode, TElCPParameters Params)
at SBPublicKeyCrypto.TElRSAKeyMaterial..ctor(TElCustomCryptoProvider Prov)
at SBPublicKeyCrypto.TElRSAKeyMaterial..ctor()
at SBClient.TElSecureClient.Init()
at SBClient.TElSecureClient..ctor(Component Owner)
at SBSimpleFTPS.TElSimpleFTPSClient..ctor(TSBComponentBase AOwner)
at SBSimpleFTPS.TElSimpleFTPSClient..ctor()
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.OBNIPPAGHDDHPOBHOGBJLEANJJPFMKFFKJEF.JDBIEFJDGLPICBBAHNBGLAMEELCOMLELLFNE(ConnectionClass , Boolean , VariableReplacerClass )
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.OBNIPPAGHDDHPOBHOGBJLEANJJPFMKFFKJEF.JDBIEFJDGLPICBBAHNBGLAMEELCOMLELLFNE(String , Boolean , VariableReplacerClass )
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.FJNDGAAMHPBCPDMHCNHGOGBCGPDLLJGOOFLF.Run()
CryptoProvBuiltIn.TElBuiltInCryptoProvider.InternalCreateKey(Int32 Algorithm, Int32 Mode, Byte[] AlgOID, Byte[] AlgParams, TElCPParameters Params)
at SBCryptoProvBuiltIn.TElBuiltInCryptoProvider.CreateKey(Int32 Algorithm, Int32 Mode, TElCPParameters Params)
at SBPublicKeyCrypto.TElRSAKeyMat

9/22/2012 12:58:00 AM Err Unhandled exception in ProcessTaskFTP.Run: System.ApplicationException: Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))
at System.Threading.ReaderWriterLock.ReleaseWriterLockInternal()
at SBSharedResource.TElSharedResource.Done()
at SBCryptoProvBuiltIn.TElBuiltInCryptoProvider.InternalCreateKey(Int32 Algorithm, Int32 Mode, Byte[] AlgOID, Byte[] AlgParams, TElCPParameters Params)
at SBCryptoProvBuiltIn.TElBuiltInCryptoProvider.CreateKey(Int32 Algorithm, Int32 Mode, TElCPParameters Params)
at SBPublicKeyCrypto.TElRSAKeyMaterial..ctor(TElCustomCryptoProvider Prov)
at SBPublicKeyCrypto.TElRSAKeyMaterial..ctor()
at SBClient.TElSecureClient.Init()
at SBClient.TElSecureClient..ctor(Component Owner)
at SBSimpleFTPS.TElSimpleFTPSClient..ctor(TSBComponentBase AOwner)
at SBSimpleFTPS.TElSimpleFTPSClient..ctor()
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.OBNIPPAGHDDHPOBHOGBJLEANJJPFMKFFKJEF.JDBIEFJDGLPICBBAHNBGLAMEELCOMLELLFNE(ConnectionClass , Boolean , VariableReplacerClass )
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.OBNIPPAGHDDHPOBHOGBJLEANJJPFMKFFKJEF.JDBIEFJDGLPICBBAHNBGLAMEELCOMLELLFNE(String , Boolean , VariableReplacerClass )
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.FJNDGAAMHPBCPDMHCNHGOGBCGPDLLJGOOFLF.Run()

9/22/2012 2:01:00 AM Err Error connecting: Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))



9/22/2012 2:20:38 AM Err Error at output to file: System.IO.IOException: The handle is invalid.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.SeekCore(Int64 offset, SeekOrigin origin)
at System.IO.FileStream.VerifyOSHandlePosition()
at System.IO.FileStream.Seek(Int64 offset, SeekOrigin origin)
at OBOKIIHBDBPMCFALPGECCIFHCDJPEDKODDGK.CNOMLBFDJANALDAEIKLKHLCGKEJIGMLIMBLK.JCOEDAOPNNOBFIMBHMNAPKOPFNGKEJDGFJFP(FileStream& , String )


Another flavor of catastrophic error is:
11/11/2012 8:31:57 PM Info Task started: SFTP z0 (description) (49136)
11/11/2012 8:31:57 PM Err Error connecting: Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))
11/11/2012 8:31:57 PM Info Task completed: SFTP z0 (description) (49136)

As this log contained more info we will investigate further and come back to you.

---

1. do you download the files locally or to a network share?
2. can you reproduce this by running the same Task again?

---

3. have you upgraded to 6.1.5 - it would be interesting as we have upgrade internal components in that version. Installing .NET 4.0 should not affect this.

---

1. do you download the files locally or to a network share?
This is actually a FTP upload task. We don't use FTP download on this server. We use SFTP and FTPS protocols in different tasks.

2. can you reproduce this by running the same Task again?
I can reproduce the error after it has happened once until we restart the service. I cannot reproduce it any give time.

3. have you upgraded to 6.1.5 - it would be interesting as we have upgrade internal components in that version. Installing .NET 4.0 should not affect this.

We have upgraded our test server to 6.1.5 and are working on creating similar loads there to try to reproduce the ftp issue. But, production servers are running 6.1.2 and we would appreciate if you can help us find a workaround or solution in this version.

Upon closer look at the error it seems to be related to loading a keys (public key authentication at connect). Do you use public key auth?

Is the key located locally or on a network drive?

---

The authentication type is "Password" for both ftps and sftp connections. We don't use public key authentication for ftp tasks.

I am beginning to think there is a problem with disk or IO controller. There are just too many errors about writing. Like this:

Internal error: Unable to successfully execute disk IO on the file system

This and the catastrophic failure are errors I have never seen before so I assume they must be very uncommon.

---

Hi Henrik,

I wanted to give you some more information on the FTP protocols that we use:

FTPS uses port 990 and public key algorithm RSA with 4096 bits.
SFTP uses port 22 and ssh dss 4096 bit certificate.

Hi Henrik,

These errors occur on FTP upload tasks so why do we get errors related to writing?

Thanks

Hi Henrik,

I've looked further into some instances of this and found some more details that might be useful.

First we get this error in the VC log:

11/19/2012 12:31:45 AM Err Failed to upload file: H:\TempFilesToFTPToIAD\dwayprod_dway_appl_20121118_0121_full.bak.z17 to /Database/MSSQLBackup_Zipped/dwayprod_dway_appl_20121118_0121_full.bak.z17, err: Error in the application.

Which corresponds with this logging from our ftp server only about a minute after it starts moving a 512mb file:

[11] Mon 19Nov12 01:30:50 - (013565) Received file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z16" successfully (409.48 KB/sec - 512,000,000 Bytes)
[11] Mon 19Nov12 01:30:51 - (013565) Receiving file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z17"
.
.
.
[11] Mon 19Nov12 01:31:45 - (013565) Error receiving file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z17", aborting (23,166,976 Bytes)
[02] Mon 19Nov12 01:31:45 - (013565) Closed session
[02] Mon 19Nov12 01:31:45 - (013565) User "dway_sched_db1" logged out

After that we don't get any logging on our ftp server from VC process until after we restart the service. All we get our the "Catastrophic Failure" error, which I take it is from the server itself failing to create a new ftp connection following the original "Error in the application." If you have ever seen that error before on ftp tasks, that could point us in the right direction of figuring out what's causing this issue.

Thanks,
Nathan

Hi Henrik,

I've looked further into some instances of this and found some more details that might be useful.

First we get this error in the VC log:

11/19/2012 12:31:45 AM Err Failed to upload file: H:\TempFilesToFTPToIAD\dwayprod_dway_appl_20121118_0121_full.bak.z17 to /Database/MSSQLBackup_Zipped/dwayprod_dway_appl_20121118_0121_full.bak.z17, err: Error in the application.

Which corresponds with this logging from our ftp server only about a minute after it starts moving a 512mb file:

[11] Mon 19Nov12 01:30:50 - (013565) Received file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z16" successfully (409.48 KB/sec - 512,000,000 Bytes)
[11] Mon 19Nov12 01:30:51 - (013565) Receiving file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z17"
.
.
.
[11] Mon 19Nov12 01:31:45 - (013565) Error receiving file "D:\ProdDisasterRecovery\Database\MSSQLBackup_Zipped\dwayprod_dway_appl_20121118_0121_full.bak.z17", aborting (23,166,976 Bytes)
[02] Mon 19Nov12 01:31:45 - (013565) Closed session
[02] Mon 19Nov12 01:31:45 - (013565) User "dway_sched_db1" logged out

After that we don't get any logging on our ftp server from VC process until after we restart the service. All we get our the "Catastrophic Failure" error, which I take it is from the server itself failing to create a new ftp connection following the original "Error in the application." If you have ever seen that error before on ftp tasks, that could point us in the right direction of figuring out what's causing this issue.

Thanks,
Nathan

Thanks for the information. I do not think this is related to the FTP server or the communication with it. Rather a local hardware problem on the server where VisualCron is installed. Have you tried running VisualCron from another server?

---