We found an issue with the FTP process within VisualCron, we rely heavily on FTP and the error handling process to function correctly within our environment. Basically during an FTP session, VisualCron reported a 'Control channel transfer error' issue, the issue we have, after this issue, VisualCron assumed the FTP completed and removed the original file. We were informed later on by our client that the data file did not transfer, thus we had to manually resend the data.

I was under the impression if the FTP process did not complete successfully, the data file would not be removed. This has been the case in other issue's, but am wondering if this is an issue when 'Control channel transfer error' happens?

Cut of the log:
...
TYPE A
200 Type set to A.

PASV
227 Entering Passive Mode (148,92,31,60,41,244)

STOR REIC_8601_009785448.DAT

STDERR:
Error uploading file(s): REIC_8601_009785448.DAT, err: Control channel transfer error

We checked our code and we do not delete any file if errors occurs in the current transfer. But we do treat each file uniquely - for example, file 1 might fail and we do not delete file A but then file 2 could success and then we delete file 2.

---

Is there a log I can send you so you can investigate, this has happened on a couple of occasions, same error.

thanks

No such logging do currently exist. We just added it for the 6.1.3 release.

We do log which files were uploaded (exists as a Variable) in current version but not which files which were deleted. You have to wait for 6.1.3 for that.

---

So would there be a way in the new version to verify successful uploads, to delete local files? Would it be an option maybe to have the process ensure the file exists on the target system, and matches before removing from the local system? This would prevent this kind of issue.

thanks

This kind of logging is what we have added to 6.1.3

---

One thing I was just thinking, we recently upgraded to version 6.1.0 from version 5.6.5, and have been receiving random ftp issue's since then. I was just given some more information from a client that the file in question will reside on the destination side, as a zero byte file, and be removed from the source side as being sent successfully.

Thought this might help.

thanks

Upon further investigation on my side, the zero byte file my client reported on, did not even show a failure on the logs, so from my point of view, everything was good, but they received a zero byte container file basically.

thanks

Hard to say what happened here, if the file really was written complete before sending?

---

I've had this happen a couple of times with a client.
It turned out that the file mask I was using was not picking up the file to send over.
It still executed the the ftp, but a zero-length file was created on the the client's side.

Check to make sure your file masks are, in fact, picking up and existing file to ftp over.
You can use the 'test' feature when filtering to make sure it can see it.

According to the log everything looked normal and good, but the destination only had a zero byte file. Does the internal process verify the file was completed, and filesize matches to the source to ensure proper sending?

thanks

According to the log everything looked normal and good, but the destination only had a zero byte file. Does the internal process verify the file was completed, and filesize matches to the source to ensure proper sending?

thanks

Currently there is no verification so the following can happen:

1. file is sent before written complete locally
2. error occurs while sending so file is not written to remote location entirely

---

File mask is *.*, with exception set to incopy.*. We have never had this issue until upgrading, and we have been running this process for some time now.

thanks

1. file is sent before written complete locally

I post the file to the scan directory with incopy.* filename, and rename when copy is completed, that would prevent attempting to send a file that is currently being written to.

1. file is sent before written complete locally

I post the file to the scan directory with incopy.* filename, and rename when copy is completed, that would prevent attempting to send a file that is currently being written to.

Yes, or you can use a File trigger which wait for release of file.

---

Running version 6.1.0

We had another FTP issue last night, where I gathered a list of files, sent all *.tmp data file, seemed to have an ftp communication issue with no notification, which stopped sending the remainder files. Thus my rename loop based on the dir listing attempted to rename files that actually didn't send until the next timed run. One thing to note, the file the process was trying to send, produced a zero byte file on the destination, and removed the local file, so we technically missed sending a file.

Log cut: **
STOR 7214_CHA-MIN28694_30533760_284803.pdf.tmp
150 Opening data connection for 7214_CHA-MIN28694_30533760_284803.pdf.tmp.

226 Transfer complete.

TYPE A
200 Type set to A; form set to N.

PASV
227 Entering Passive Mode (159,249,160,185,157,229)

STOR 7214_CHA-MIN29340_30533710_285098.pdf.tmp
150 Opening data connection for 7214_CHA-MIN29340_30533710_285098.pdf.tmp.

226 Transfer complete.

TYPE A
200 Type set to A; form set to N.

PASV
227 Entering Passive Mode (159,249,160,185,157,230)

STOR 7214_CHA-TM29226_30533761_285081.pdf.tmp

STDERR:


9/20/2012 9:17:54 PM

As you can see, after the STOR, there is nothing, previous sends, there is transfer complete. And no error found in STDERR

Rename failure: **
RNFR /ecom/ecadmin/G2P/7214_CHA-TM29226_30533761_285081.pdf.tmp
450 /ecom/ecadmin/G2P/7214_CHA-TM29226_30533761_285081.pdf.tmp: Cannot open or remove a file containing a running program.


STDERR:
Error renaming a FTP file: /ecom/ecadmin//7214_CHA-TM29226_30533761_285081.pdf.tmp, err: Unaccepted server reply (error code is 450) (450 /ecom/ecadmin/G2P/7214_CHA-TM29226_30533761_285081.pdf.tmp: Cannot open or remove a file containing a running program.
)

9/20/2012 9:17:56 PM

But there is no Transfer complete after:

STOR 7214_CHA-TM29226_30533761_285081.pdf.tmp

The error 450 means simply:

File unavailable (e.g., file busy). Try again later.

---

Correct, basically the initial send of the file was un-successful, but VisualCron assumed it was I'm guessing as the source file was removed. As for the rename, yes I understand there was an issue, as the file was not actually sent successfully.

Correct, basically the initial send of the file was un-successful, but VisualCron assumed it was I'm guessing as the source file was removed. As for the rename, yes I understand there was an issue, as the file was not actually sent successfully.

So, the problem is that a file was created even though it failed?

---

Two issue's here, on the destination, a container file was created, zero byte, that's not the big issue here. The main issue we had was VisualCron assumed the file was sent, and removed the file from the local disk. Thus delaying this data getting to the destination, until our client notified us. What we need, is if the file did not successfully send, produce error, and leave the data file on the disk, for out next FTP session to attempt sending.

thanks

Just to verify. How does the rename come into picture and how is that renamed connected to the download?

I looked at the code for download and I see no way that it deletes the file if it has not been successfully downloaded. I can understand the the rename fails but I do not see how it is connected to the delete of the file?

---

The process I needed to setup was as follows, since there currently is no temp file upload, or rename in the FTP upload option:

- Rename all files in local directory with *.tmp
- Gather file listing of all *.tmp files
- FTP all *.tmp files to destination, deleting after successful send.
- Log FTP session to file
Loop through file listing results:
- Rename *.tmp file by removing .tmp extension
- Log rename to file

By the rename point, the local file is gone already.

thanks

I see, I thought this was a download. We have actually made changes in this. Do not know if it was in 6.1.1. or 6.1.2. however I suggest you download the latest.

---

I will look into upgrading, unfortunately it's not that simple, I have to go through approvals and such, since this is a production server. Hopefully I can get it fast tracked.

Can you give me a list with explanations to what was changed since 6.1.0 in the FTP comms?

thanks

Previously, we deleted only if no error had occurred during transfer of a file. Now we have added an extra check that file has been uploaded. Still, we are unsure how it could delete and not throw an error. But this is the change we have made. Besides that, we have added some extra logging concerning this.

---