Backup Settings Jobs Fail - VisualCron

Please note that VisualCron support is not actively monitoring this community forum. Please use our contact page for contacting the VisualCron support directly.

Welcome Guest! To enable all features please Login or Register.

keithdavis
Free support Topic Starter

2013-08-29T05:13:55Z

I have one machine that has worked fine for years, but suddenly fails on both it's Backup Settings jobs. One job is the default Backup Settings job, the other is a job that stores the backups in a network location.

We changed the Service Account and now it fails (with or without credentials.) I uninstalled and re-installed VisualCron and it worked fine for a few days, but now it fails again. Both jobs report the same Output(stderr):

Exception in Task: Could not find a part of the path 'C:\Users\networkservice\AppData\Local\Temp\KDAVIS.zip.~'.

That user, "networkservice" is the service account. However, even with Credentials entered (not the same user, keithdavis), I get the exact same error.

Edited by moderator 2013-10-03T07:21:11Z | Reason: Not specified

Sponsor

Forum information

Wanna join the discussion?! Login to your Forum forum account or Register a new forum account

Support
Official support

2013-08-30T07:59:10Z

I would recommend running as SYSTEM. Is this folder existing:

C:\Users\networkservice\AppData\Local\Temp

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

keithdavis
Free support Topic Starter

2013-08-30T13:36:36Z

The second backup job and most of our other jobs require admin credentials to access network resources. We switched from using Credentials in VisualCron to using the credential as the Logon As for the Service because managing Credentials when moving or copying 200 jobs on 12 servers is too unwieldy and we can set the Service account using Group Policy.

Ok, so I checked and that folder does not exist, however, I checked and that folder did not exist once before and I created the last time this failed, but now it's gone again. Why would it keep disappearing? Why does the job ignore my Credentials if I enter them when the service is not System.

The whole Credentials or what user the jobs run as is very inconsistent (see my other posts, like with HTTP.)

We really need a way to manage all of this globally. Most of our servers run duplicates of the same jobs and this takes up way to much time to manage.

I did successfully update a bunch of jobs using C# and the API the other day, which is promising.

Back to topic at hand, another issue with this same machine is that at times it won't connect on opening the client, I get and AD Login failed message. If I click Connect manually, it works fine.

keithdavis
Free support Topic Starter

2013-08-30T13:38:49Z

Sorry, wrong about the error message and it's happening every time now. The last time I fixed this by reinstalling, but it only lasted a week..

Error Message:

Login failed, reason: Unhandled AD error occurred. Please view server log file.

From the log file:

8/30/2013 8:38:02 AM Err Logon error for AD user: 'keithdavis' - unhandled AD error: Illegal operation attempted on a registry key that has been marked for deletion.

keithdavis
Free support Topic Starter

2013-08-30T13:41:44Z

Restarted service, had to close and re-open the client (was getting RPC failure) and now it connects and the backup job works.

Very frustrating.

Why doesn't VisualCron know to create that folder if I does not exist?

KJDavie
No customer

2013-09-02T01:30:27Z

Hi Keith,

We have seen Backups stop working (only once, and not repro'd so I didn't raise).

We are on 7.0.1, and this error occurred with that version (1 server only).

If it occurs again I will try to catch more comprehensive details and log it.

Also with Backups, we only do this Operation locally, to local disk, and then push them (and the Logs and the Output Database) to our NAS as a separate, standard VisualCron File Copy task from each server (with a Credential).

--- *** ---

A bit of a tangent, and not directly on your issue . . .

How long do you run VisualCron Server/Service for, do you ever restart it other than diagnosis / Server restarts associated with other activities ?

You may have already covered this off, but as you are multi-server VC site, and if you have your firewall rules <16444> set for another server to control, can you try planned, scheduled, Visualcron service restarts as a preventative measure ? <you won't be able to do this as a job on the local server>.

We have found *run to failure* be an unreliable approach for a number of Windows Server apps, and we don't find Visualcron any different.

A common diagnosis technique is to restart the service (not just VisualCron).

We have some other Apps also that exhibit memory leakage over time.

We only restart the servers when necessary, but Services related to a number of our Key Apps are restarted periodically during a scheduled blackout period to deal with any number of ills.

This obviously forces some housekeeping with VC itself (shutdown, startup), cleans up any leakage, and also re-establishing any external links (such as AD / Network Shares etc).

The downside <as you noted> is active Clients (if left Open) will alert and need to reconnect.

We believe taking the periodic restart at a time of our choosing is preferable to an unplanned one (at potentially a high load time), and we *believe* we see more reliability in the times we care about by doing this at a time we don't.

Good Luck !

Kevin

Edited by user 2013-09-02T01:44:57Z | Reason: Wording Massage

KJDavie attached the following image(s):

20130902 - Push Server Backups Remote onto NAS.png

20130902 - VisualCron Service Restarts - Done on Remote Servers.png

keithdavis
Free support Topic Starter

2013-09-03T17:00:20Z

Originally Posted by: KJDavie

Hi Keith,
How long do you run VisualCron Server/Service for, do you ever restart it other than diagnosis / Server restarts associated with other activities ?

You may have already covered this off, but as you are multi-server VC site, and if you have your firewall rules <16444> set for another server to control, can you try planned, scheduled, Visualcron service restarts as a preventative measure ? <you won't be able to do this as a job on the local server>.

We have found *run to failure* be an unreliable approach for a number of Windows Server apps, and we don't find Visualcron any different.

We don't restart the VC service, except for troubleshooting / repair.

Henrik, do you have any comment on this protocol?

If we feel a service has to have that protocol in place, we'll do it, but in general it's a last resort.

The servers all get restarted from time to time (Windows Updates, etc...)

keithdavis
Free support Topic Starter

2013-09-10T16:27:17Z

FYI, I just added a task to create the folder C:\Users\networkservice\AppData\Local\Temp before the backup settings task.

Support
Official support

2013-09-11T07:02:40Z

Originally Posted by: keithdavis

Restarted service, had to close and re-open the client (was getting RPC failure) and now it connects and the backup job works.

Very frustrating.

Why doesn't VisualCron know to create that folder if I does not exist?

There are a couple of problems with this;

1. we cannot assume that you *really* want this folder - it might be an input error and we show this instead
2. any "auto-creation" of folders would need extra code. We are talking of a lot of Tasks that work with folders and the question is if we should implement for all

But again, I do not think we can assume that you really want this folder. Auto-creating folders can create a lof of different behavior.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

Support
Official support

2013-09-11T07:04:31Z

#10

Originally Posted by: keithdavis

Originally Posted by: KJDavie

Hi Keith,
How long do you run VisualCron Server/Service for, do you ever restart it other than diagnosis / Server restarts associated with other activities ?

You may have already covered this off, but as you are multi-server VC site, and if you have your firewall rules <16444> set for another server to control, can you try planned, scheduled, Visualcron service restarts as a preventative measure ? <you won't be able to do this as a job on the local server>.

We have found *run to failure* be an unreliable approach for a number of Windows Server apps, and we don't find Visualcron any different.

We don't restart the VC service, except for troubleshooting / repair.

Henrik, do you have any comment on this protocol?

If we feel a service has to have that protocol in place, we'll do it, but in general it's a last resort.

The servers all get restarted from time to time (Windows Updates, etc...)

I am not sure I understand the question "Henrik, do you have any comment on this protocol?"?

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

Support
Official support

2013-09-11T07:06:26Z

#11

This post is a bit unclear as it is now. What is the problem right now?

1. you should use SYSTEM - not network service
2. you should use a Credential with local logon unchecked if saving to network share. If you are not doing that (saving or reading from network share) - then don't use a Credential.

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

Support
Official support

2013-09-11T07:18:33Z

#12

Originally Posted by: keithdavis

FYI, I just added a task to create the folder C:\Users\networkservice\AppData\Local\Temp before the backup settings task.

I see now what you mean.

We have added logging for next version to confirm what temp path is really used and if it exists. Look for release later today in beta for 7.0.5.

Edited by user 2013-09-11T07:19:07Z | Reason: Not specified

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

Support
Official support

2013-09-11T10:17:15Z

#13

Please test latest beta now:

http://www.visualcron.co....aspx?g=posts&t=3339

We have added logging regarding to what folder is being used for temporary folder. We have also tried to explicity set the temporary folder in that Task (with the settings folder specified in Server settings->Folders).

Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

keithdavis
Free support Topic Starter

2013-09-11T14:57:38Z

#14

Originally Posted by: Support

This post is a bit unclear as it is now. What is the problem right now?

1. you should use SYSTEM - not network service
2. you should use a Credential with local logon unchecked if saving to network share. If you are not doing that (saving or reading from network share) - then don't use a Credential.

FYI, we are not using the "network service" service account, we are using an Administrator account called Network Service (domain admin) as the service account.

We don't want to use credentials anymore, because it's too hard to manage on so many jobs and servers. This configuration (using the domain admin account as the service account), works fine on 10 other servers.

keithdavis
Free support Topic Starter

2013-09-11T15:10:30Z

#15

Originally Posted by: Support

Please test latest beta now:

http://www.visualcron.co....aspx?g=posts&t=3339

We have added logging regarding to what folder is being used for temporary folder. We have also tried to explicity set the temporary folder in that Task (with the settings folder specified in Server settings->Folders).

Installed 7.05 beta. Disabled the create temp folder task for Backup settings. The location is completely different!

C:\Program Files (x86)\VisualCron\backup\VC-Settings.zip

Wait, that's the output. Anyway, I also deleted the Temp folder and ran the job, Temp folder not re-created, but no failure either.

Edited by user 2013-09-11T15:11:25Z | Reason: Not specified

Support
Official support

2013-09-11T16:16:44Z

#16

Originally Posted by: keithdavis

Disabled the create temp folder task for Backup settings./quote]

What kind of Task or setting was that? So everything works now?
Henrik
Support
http://www.visualcron.com
Please like VisualCron on facebook!

keithdavis
Free support Topic Starter

2013-09-11T18:21:46Z

#17

Originally Posted by: Support

What kind of Task or setting was that? So everything works now?

I had added a Task that created the Temp folder before the Backup settings job ran. I disabled that Create Folder Task, and with 7.0.5, it appears to be working.