could be variable or ftp download problem - VisualCron

Please note that VisualCron support is not actively monitoring this community forum. Please use our contact page for contacting the VisualCron support directly.

Welcome Guest! To enable all features please Login or Register.

ErikC
Free support Topic Starter

2008-11-24T08:37:18Z

Hi,

I have a job which runs every 30 minutes (interval 30 minutes at xx:02:33 and xx:32:33).
It has 5 tasks in it.
1. It deletes all files (Delete file) in a temporary folder
2. It creates a file (Write file) with one line of static text and the filename is {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt and put it in a different folder than step 1.
3. It waits (Wait) 2 minutes
4. It downloads a file (FTP/FTPS/SFTP) from a ftp site and it is downloading the {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt and places this file in the temp folder of step 1.
5. This task may run if the {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt file exist (condition File exist) is False in the temp folder. than it is sending an email (Email) telling me the file is not retrieved.

When step 2 happens, there is an other job ftp-ing the file to an ftp site. That's why I have a wait for 2 minutes so there is a lot of time to do this.

In my logs I can see that the upload of the other jobs works fine. The file is ate the other site.
Funny thing is that I receive emails that the file is not there. Why? The file is not downloaded, but not all the time.

Example:
file 200811241232AM.txt upload ok, download ok, no email
file 200811240102AM.txt upload ok, download nok, email send
file 200811240132AM.txt upload ok, download ok, no email
file 200811240202AM.txt upload ok, download nok, email send
etc

Here are more failure times:

22-11 1:32 AM
22-11 2:32 AM
22-11 4:32 AM
22-11 5:32 AM
22-11 6:32 AM
22-11 8:32 AM
22-11 9:32 AM
22-11 10:32 AM
22-11 11:32 AM
22-11 12:32 AM
22-11 1:32 PM
22-11 2:32 PM
22-11 3:32 PM

23-11 7:02 PM
23-11 8:02 PM
23-11 9:02 PM
23-11 10:02 PM
23-11 11:02 PM

24-11 1:02 AM
24-11 2:02 AM
24-11 3:02 AM
24-11 4:02 AM
24-11 5:02 AM
24-11 6:02 AM
24-11 7:02 AM
24-11 12:02 AM

I double checked the variable {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt in each task, so there are no typo's in there.

Whay is it failing somitimes?

Currently this is on a 4.9.38 VC server.

Regards,
Erik

Edited by user 2008-11-25T08:10:40Z | Reason: Not specified

Uses Visualcron since 2006.

Sponsor

Forum information

Wanna join the discussion?! Login to your Forum forum account or Register a new forum account

Support
Official support

2008-11-24T09:41:23Z

1. Is the file really in the temp folder?
2. What is the last modified time of the file in the temp folder?

I suspect that either no file enters the tmp-folder or it is to late. I see no reason for a file exist check to fail.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T09:47:08Z

If I receive an email, the log of the download shows me there is no file downloaded. The condition works fine at that moment and starts the email task.
The strange thing is that the download file task sometimes downloads the file and sometime it doesn't.
The problem might be over here.

Edited by user 2008-11-24T10:04:16Z | Reason: Not specified

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T10:08:23Z

The FTP session is a FTP TLS Implicit (port 990) connection type: PORT

Failing download log:

Citrix Portal secure FTP server
USER profnet_upload
Password required for profnet_upload
PASS ****
Logged on
PROT P
Protection level set to P
PWD
"/" is current directory.
CWD /TEST/Filetransfer
CWD successful. "/TEST/Filetransfer" is current directory.
PWD
"/TEST/Filetransfer" is current directory.
TYPE A
Type set to A
PORT 10,5,102,53,15,104
Port command successful
LIST
Opening data channel for directory list.
Transfer OK
QUIT

Good download log:

Citrix Portal secure FTP server
USER profnet_upload
Password required for profnet_upload
PASS ****
Logged on
PROT P
Protection level set to P
PWD
"/" is current directory.
CWD /TEST/Filetransfer
CWD successful. "/TEST/Filetransfer" is current directory.
PWD
"/TEST/Filetransfer" is current directory.
TYPE A
Type set to A
PORT 10,5,102,53,13,224
Port command successful
LIST
Opening data channel for directory list.
Transfer OK
PWD
"/TEST/Filetransfer" is current directory.
PWD
"/TEST/Filetransfer" is current directory.
TYPE A
Type set to A
PORT 10,5,102,53,13,225
Port command successful
LIST
Opening data channel for directory list.
Transfer OK
TYPE I
Type set to I
PORT 10,5,102,53,13,226
Port command successful
RETR 200811240932AM.txt
Opening data channel for file transfer.
Transfer OK
DELE 200811240932AM.txt
File deleted successfully
QUIT

Edited by user 2008-11-24T10:11:15Z | Reason: Not specified

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T10:31:02Z

I will try the 'Use raw linsting fallback' may be this solves the error....

Uses Visualcron since 2006.

Support
Official support

2008-11-24T10:34:00Z

Raw listing is for servers that does not send proper listings. It either works or not - it does not work some time and sometimes not. So, if it just fails some time the file is probably not there when trying to download.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T10:38:02Z

hmm, okay.. I just received an email telleing me the file is not received, so the raw was indeed not the answer. I will add a dir listing task to see what files are there before I do the download.

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T11:03:00Z

My job started at 10:50AM

The upload went file:

PUTFILE("\\server103\data\Theodoor Gilissen Services\Profnet Postbus\TEST\FileTransfer\200811241050AM.txt")
Uploading.................		200811241050AM.txt
OK

The files at the remote site:

CHDIR("/test/filetransfer")
Changing current remote directory to /test/filetransfer

GETLIST(REMOTE_FILES)
Getting file listing of current remote directory
Found 1 file.
200811241050AM.txt

The download log is:

Citrix Portal secure FTP server
USER profnet_upload
Password required for profnet_upload
PASS ****
Logged on
PROT P
Protection level set to P
PWD
"/" is current directory.
CWD /TEST/Filetransfer
CWD successful. "/TEST/Filetransfer" is current directory.
PWD
"/TEST/Filetransfer" is current directory.
TYPE A
Type set to A
PORT 10,5,102,53,17,79
Port command successful
LIST
Opening data channel for directory list.
Transfer OK
QUIT

And I got my email again. So to me it looks that the download tast is not doing his thing. The filemask is: {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt
It could be that the variable is failling in the download mask?

Regards
Erik

Edited by user 2008-11-24T11:04:17Z | Reason: Not specified

Uses Visualcron since 2006.

Support
Official support

2008-11-24T11:35:59Z

If nothing is downloaded the file does not exist or the download mask is wrong. Are you trying to use the exact mask or wild card?

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T11:37:54Z

#10

ErikC wrote:
The filemask is: {JOB(Active,LastRun,yyyyMMddhhmmtt)}.txt

Uses Visualcron since 2006.

Support
Official support

2008-11-24T12:36:46Z

#11

I would log the filemask to a file before downloading. If the mask is OK the file is probably not present at the server at download time.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T12:48:19Z

#12

I will log the filename to a file. In my previous reply you could see the remote file was present before downloading.

Uses Visualcron since 2006.

Support
Official support

2008-11-24T12:56:48Z

#13

I see. But don't you think it is a great risk that you have to match the file by the minute? It is very easy that the time on the server is not the same as on the windows machine - also, delays may cause diskrepancy in time. Why don't you download all the files to a folder that has a file trigger instead?

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T13:04:18Z

#14

What I wanted to do is to upload a file with a unique name. So I used the job run time for this. The job runs every 30 minutes, so the fiename differs each time.
All tasks in the job could use this variable of the runtime of the current job and getting the same time. It is not that I get the time from a different server, it is the time the job started.
So in my opinion it should work, also a wait should not matter. The job time is in every task within the job the same.

I will try a download of all files. The condition should see if the file I want is there or not.

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T13:16:41Z

#15

I tried the *.* filemask for downloading.

It is not working as it should be. There are no files downloaded and there is one at the other site.
This happens not all the time though, I tried it several times now and sometimes the file is downloaded, and sometimes the file isn't.

There is nothing fancy for the download task, just downloading all files from a directory.

At this moment it looks that the download file task is sometimes working and sometimes it is not downloading anything.
I will try to download the file with a different FTP client and will let you know the result.

Uses Visualcron since 2006.

Support
Official support

2008-11-24T13:27:29Z

#16

Yes, it looks like the timing is very important and maybe hard to debug. When testing with another client you need to refresh the folder the same second as VC runs the download Task.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-11-24T13:32:04Z

#17

I'm using an other commandline ftp client for this (scriptftp) and I will replace the current download task with this one (or disable the VC one) so it is in the same spot.

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T14:11:07Z

#18

Strange things happen over here. I tried a different ftp client and this works fine. I enabled the VC job and both task do the same. Download everything. Now I removed the scriptftp job leaving me with the previous tasks and it works also for the last 20 runs (I reduced the trigger time to a few seconds)

Edited by user 2008-11-25T07:53:48Z | Reason: Not specified

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-24T14:31:43Z

#19

and it failed again... I will let the other ftp script run for a day to see if this is working. Let you know the results.

Regards
Erik

Uses Visualcron since 2006.

ErikC
Free support Topic Starter

2008-11-25T08:01:56Z

#20

This morning I checked the VC job and every run (30 minutes interval) the external ftp process downloaded the file I uploaded. So no problems with an external ftp client.
What could this be Henrik? Is the download task not functioning the way it should be?

Edited by user 2008-11-25T08:02:28Z | Reason: Not specified

Uses Visualcron since 2006.

Support
Official support

2008-11-25T09:11:10Z

#21

I don't see how, but we have to try reproducing it here.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

erich
No customer

2008-12-12T21:22:28Z

#22

We're having a very similar problem with a FTP-TLS task (explicit, though, not implicit). Even when we use a *.* filemask, and ensure that files are present to be downloaded, we get nothing.

Unlike your issue though, this happens every single time. We've already ruled out firewall or permissions errors - on the same PC with the same credentials I can use FileZilla to browse and download files just fine.

Support
Official support

2008-12-13T02:45:31Z

#23

Erich, do you have any test account we could use (support@visualcron.com)?

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

ErikC
Free support Topic Starter

2008-12-22T14:57:46Z

#24

This log is from my external ftp client:


Connecting to ipnumber
Resolving host name "ipnumber"
Connecting to ipnumber Port: 990
Connected to ipnumber.
Connected. Exchanging encryption keys...
Session Cipher: 128 bit RC4
TLS encrypted session established.
220 Portal seure FTP server
USER xxxxx
331 Password required for xxxxx
PASS XxXxXx
230 Logged on

...

SYST
FEAT
OPTS UTF8 ON
PWD
CWD /
PWD
TYPE I
PBSZ 0
PROT C
PORT i,p,n,u,m,b,e,r,8,226
MLSD
PORT i,p,n,u,m,b,e,r,8,226
STOR filename

The VisdualCron log:

Portal seure FTP server
USER xxxxxx
Password required for xxxxxx
PASS xXxXxX
Logged on
PROT P
Protection level set to P
PWD
CWD /
TYPE A
PORT i,p,n,u,m,b,e,r,7,171
LIST
TYPE I
PORT i,p,n,u,m,b,e,r,7,172
STOR filename

The SYS and the FEAT are just information about the server.
The OPTS UTF8 ON forces to use UTF8.
MLSD is also for nice response information.
But there is a difference in the PROT command.
VC is using protection level P as the external one is using C.

Could this be the issue?

Regards
Erik

Edited by user 2008-12-22T15:23:23Z | Reason: Not specified

Uses Visualcron since 2006.

Support
Official support

2008-12-22T15:09:36Z

#25

We could create a version with optional sending of PROT P to test. But in your case it seems like STORE was sent - was not that transfer a success?

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!