I have a Workflow Manager and Service Bus farm configured and working. I am trying to add a second server to the farm. During configuration, as the second host is being added through Add-SBHost, there is an attempt made to stop and start Service Bus services on the first host. The start fails because the Service Bus Message Broker service gets stuck in the "Starting" state. It retries this 3 times, each try taking around 7 minutes and each time the failure ending with a message that looks like this:
ServiceBus Broker service failed to start, retry count 1. Exception message: Operation timed out.. Stack Trace: at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)
at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()
at Microsoft.Cloud.HostingModel.ComponentHost.OpenComponent(IComponent component, RequestTracker tracker)
at Microsoft.Cloud.HostingModel.ComponentHost.Open()
at Microsoft.ServiceBus.MessageBroker.Backend.OnStart(String[] args)
This error is preceded by 2 others that present no more information than the above does:
TrackingId: f8ff41e9-53e0-48fc-b4de-b7505d4d5f7b, SubsystemId: NoSystemTracker. Failed to open Messaging Host Component System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF
at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)
at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)
at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)
--- End of inner exception stack trace ---
at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)
at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()
and...
Windows Fabric Runtime create failed Exception System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF
at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)
at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)
at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)
--- End of inner exception stack trace ---
at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)
at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
The Windows Fabric Runtime creation itself seems to be retried 6 times, each try taking around 70 seconds and each time it fails, the following information message is posted to the log:
Windows Fabric Runtime create failed (Retries left 6) Exception System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF
at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)
at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)
at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)
--- End of inner exception stack trace ---
at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)
at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
Can anyone throw some light on what could be causing this failure? I am hoping the answer is not uninstall everything and reinstall.