We have a problem where the Visual Cron Service (8.1.1 on a Windows 2012 server fully up to date) is crashing every few days at different times. The same process seems to have run prior each time but it is frequently run process so this could be a coincidence. It started happening after we added a task that used the ZIP method but we have disabled that task and it continues to occur.
I thought it might the ZIP task because of these forum posts showing very similar reports to what is happening with us:
The crashing has continued despite this.
Visual Cron does not come back up by itself after a crash and when we manually restart the service without a reboot the port it listens on is bound so we cannot connect using remote clients (we can RDP in and use the server normally). If we were the change the listening port in server settings it can bind that new port and remote clients can connect.
I can provide you with the dump files and any other information (e.g. configuration backup, logs, etc) that you might require but I do not want to post those in your public support forum.
Here is an info dump on the problem with specifics:2016-07-04 9:48:31
Application: VisualCronService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
Stack:
at VisualCronAPI.ClientProcessWorkerThread.Run()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()
2016-07-04 9:48:32
Faulting application name: VisualCronService.exe, version: 8.1.1.18743, time stamp: 0x57441de0
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007ffb2445db2c
Faulting process id: 0x6b0
Faulting application start time: 0x01d1d202fdb1a0cd
Faulting application path: C:\Program Files (x86)\VisualCron\VisualCronService.exe
Faulting module path: unknown
Report Id: 9c7e29be-41e5-11e6-80cb-b083fedc044a
Faulting package full name:
Faulting package-relative application ID:
2016-07-04 9:49:50
Last Visual Cron log file entry in log_server20160704.txt until restart:
2016-07-04 9:49:23 AM Info Task started: Hit schedule URL (3061734)
2016-07-04 9:49:23 AM Debug Executing HTTP with parameters:
2016-07-04 9:49:23 AM Info Task completed (Success)->'Hit schedule URL ' (3061734)
2016-07-04 9:49:23 AM Info Job completed (Success)->'Daily Schedule'
2016-07-04 9:49:42 AM Info JobAPI->Save->Saving Jobs (124) - in shut down: False
More information about this process is below.
2016-07-04 9:49:50
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: VisualCronService.exe
P2: 8.1.1.18743
P3: 57441de0
P4: VisualCronAPI
P5: 1.0.3.18725
P6: 57441dbc
P7: 4bf
P8: ab
P9: System.NullReferenceException
P10:
Attached files:
... dump files, etc...
2016-07-04 9:49:53
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: VisualCronService.exe
P2: 8.1.1.18743
P3: 57441de0
P4: VisualCronAPI
P5: 1.0.3.18725
P6: 57441dbc
P7: 4bf
P8: ab
P9: System.NullReferenceException
P10:
Attached files:
... dump files, etc...
Visual Cron Server Log Entries on RESTARTAfter this point the Visual Cron service is stopped. When I bring the server back up the port cannot be bound (note: we have changed it from standard port to a new port as this allows the clients to connect). The server runs normally now with the exception of remote clients not connecting. The server may go down again over the course of 1-5 days in our experience.
2016-07-04 11:13:32 AM Debug Starting CommServers
2016-07-04 11:13:32 AM Debug ComService (WCFService) registered in: net.pipe://0.0.0.0/VisualCron/
2016-07-04 11:13:34 AM Debug ComService (WCFService) registered in: net.tcp://0.0.0.0:16555/
2016-07-04 11:13:37 AM Debug ComService() exception (System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint 0.0.0.0:16555. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations. ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.ServiceModel.Channels.SocketConnectionListener.Listen()
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SocketConnectionListener.Listen()
at System.ServiceModel.Channels.ConnectionAcceptor.StartAccepting()
at System.ServiceModel.Channels.ExclusiveTcpTransportManager.OnOpen()
at System.ServiceModel.Channels.TransportManager.Open(TransportChannelListener channelListener)
at System.ServiceModel.Channels.TransportManagerContainer.Open(SelectTransportManagersCallback selectTransportManagerCallback)
at System.ServiceModel.Channels.TcpChannelListener`2.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at CommServer.Classes.WCF.WCFService.Start()
at CommServer.Classes.Server.Initialize(ServerConfiguration configuration)) stopped.
2016-07-04 11:13:37 AM Debug CommServer started: 0.0.0.0:16555
I believe this is because there is a zombie process binding that port after VC has crashed...
THE PROCESS running prior to crashThe process running prior to the crash uses an HTTP task to hit an internal URL that is up and functioning. It is basically a web hook that runs processes on an internal server. The logging indicates this task runs normally (VC and our internal logging for that server, etc). What HTML response is logged by VC looks identical to the other HTML responses (no indication this is a HTTP error, etc).
Do you have any idea what could be causing the system to crash?
Edited by moderator
2016-07-05T07:55:21Z
|
Reason: Not specified