Hi.
I have a workflow that waits for an incoming notification before running a custom code activity. Once the notification is received and the custom code is called, if the custom code activity faults with a NullReferenceException then Workflow Manager proceeds
to report a number of error and critical events to the EventLog (see below) before crashing (with crash dump) and restarting. On restart, Workflow Manager attempts to resume the workflow by calling the faulting code. It will keep doing this until the crash
dumps (each approx 360MB) consume all remaining disk space; at a rate of one crash dump every 45 seconds. The only way to break the cycle before disk space is exhausted is to delete the published workflow.
To rule-out any additional interaction within the code, I pared-down my custom activity to a simple "throw new NullReferenceException" in the Execute() body and got the same result.
Incidentally, the crash dumps are written to "C:\Windows\System32\%LOCALAPPDATA%\CrashDumps" which is not a location where I would expect to find crash dumps for a service.
Regards,
Phil
Event #1 (many instances):
Dispatcher encountered an unexpected exception: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)
Event #2 (Critical):
Dispatcher failfast is causing the Workflow Manager backend to terminate. Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.NullReferenceException:
Object reference not set to an instance of an object.
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<---
Event #3:
Application terminating due to FailFast. Message : An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for
other failures.'., Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)
at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<---
.