Hello everyone,

I have been running a SharePoint 2013 farm with Workflow Manager for over 2 years now with no issues, but as of today, all the SharePoint 2013 workflows stopped working. When i try to view a workflow attached to a list, I get An Unexpected Error Has Occurred, but the ULS logs show me the following error:

Current identity context: '{"elevated":"true","nameid":"."}'e77f869d-19c6-0030-7223-53a406bb4c36
Created OAuth2 bearer credentials: {"claims":{},"_isM4ElevationBehaviorEnabled":"true"}e77f869d-19c6-0030-7223-53a406bb4c36
Authenticating OAuth2 Bearer challenge.
Retrieving the authentication realm for context '00000000-0000-0000-0000-000000000000'.
Retrieved the authentication realm for context '00000000-0000-0000-0000-000000000000' as '0216aeaa-8a46-461e-9c03-1cef2c6c9300'.
Using elevated OAuth2 bearer credentials.
Issuing OAuth2 S2S token for identity '00000005-0000-0000-c000-000000000000/WORKFLOWSERVER:12291@0216aeaa-8a46-461e-9c03-1cef2c6c9300'. tokenType: 2
Getting OAuth2 actor token for identity '00000005-0000-0000-c000-000000000000/WORKFLOWSERVER:12291@0216aeaa-8a46-461e-9c03-1cef2c6c9300'.
Token request correlation id: 'df7f869d-a9ed-0030-7223-5281f54c35c0'.
OAuth2 S2S Bearer Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IlJqeW9pNGdHLUxKZE1DaHVTMGpSdXN0X0xaVSJ9.eyJhdWQiOiIwMDAwMDAwNS0wMDAwLTAwMDAtYzAwMC0wMDAwMDAwMDAwMDAvbXRhc2hhcmVwdHdmZTAyOjEyMjkxQDAyMTZhZWFhLThhNDYtNDYxZS05YzAzLTFjZWYyYzZjOTMwMCIsImlzcyI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMEAwMjE2YWVhYS04YTQ2LTQ2MWUtOWMwMy0xY2VmMmM2YzkzMDAiLCJuYmYiOiIxNDY2MDIzNjI3IiwiZXhwIjoiMTQ2NjE1MzIyNyIsIm5hbWVpZCI6IjAwMDAwMDAzLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMEAwMjE2YWVhYS04YTQ2LTQ2MWUtOWMwMy0xY2VmMmM2YzkzMDAifQ
Application error when access /_layouts/15/Workflow.aspx, Error=The remote server returned an error: (401) Unauthorized.   at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)e77f869d-19c6-0030-7223-53a406bb4c36
System.Net.WebException: The remote server returned an error: (401) Unauthorized.    at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)e77f869d-19c6-0030-7223-53a406bb4c36
Getting Error Message for Exception System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> Microsoft.Workflow.Client.AuthenticationException: Authentication Failed. Valid credentials must be provided for one of the following protocols: Bearer, Negotiate. HTTP headers received from the server - ActivityId: 1c52e531-cf1f-43d2-a41c-f7a29aed7e15. NodeId: MTASHAREPTWFE02. Scope: /SharePoint/default/570cf36c-5364-4e0d-91c2-b0f7e1463cfb/8cba2aa0-b7f9-4652-ab71-cd9775189cc6. WWW-Authenticate: Bearer realm="0216aeaa-8a46-461e-9c03-1cef2c6c9300", client_id="00000005-0000-0000-c000-000000000000", trusted_issuers="00000003-0000-0ff1-ce00-000000000000@c72f8383-a0a5-456d-81e5-c38b124f5599", scope="/SharePoint/default/570cf36c-5364-4e0d-91c2...e77f869d-19c6-0030-7223-53a406bb4c36
...-b0f7e1463cfb/8cba2aa0-b7f9-4652-ab71-cd9775189cc6", error="invalid_token", error_description="The JSON web token's X.509 certificate thumbprint 'Rjyoi4gG-LJdMChuS0jRust_LZU' did not match any known trusted issuer public key certificates.",Bearer realm="0216aeaa-8a46-461e-9c03-1cef2c6c9300", client_id="00000005-0000-0000-c000-000000000000", trusted_issuers="00000003-0000-0ff1-ce00-000000000000@c72f8383-a0a5-456d-81e5-c38b124f5599", scope="/SharePoint/default", error="invalid_token", error_description="The JSON web token's X.509 certificate thumbprint 'Rjyoi4gG-LJdMChuS0jRust_LZU' did not match any known trusted issuer public key certificates.",Negotiate,Negotiate,NTLM. Client ActivityId : e77f869d-19c6-0030-7223-53a406bb4c36. ---> System.Net.WebException: The remote server returned an erro...e77f869d-19c6-0030-7223-53a406bb4c36
...r: (401) Unauthorized.     at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)     --- End of inner exception stack trace ---     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)     at Microsoft.Workflow.Client.InstanceManager.GetInternal(Int32 skip, Int32 count, String workflowName, WorkflowInstanceStatus workflowStatus, IDictionary`2 activationMetadataFilter)     at Microsoft.SharePoint.WorkflowServices.FabricWorkflowInstanceProvider.EnumerateByMonitoringParameter(Guid monitoringParameter, Int32 offset, Int32 count, Boolean checkPermis...e77f869d-19c6-0030-7223-53a406bb4c36
...sions, Nullable`1 secondaryMonitoringParameter)     at Microsoft.SharePoint.WorkflowServices.FabricWorkflowInstanceProvider.EnumerateInstancesForListItem(Guid listId, Int32 itemId, Int32 offset)     at Microsoft.SharePoint.WorkflowServices.FabricWorkflowInstanceProvider.EnumerateInstancesForListItem(Guid listId, Int32 itemId)     at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WorkflowPageBase.ConstructStatusArraysWF4(ArrayList running, ArrayList completed, Boolean onlyMyWorkflows)     at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WorkflowPage.ConstructStatusArrays()    at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WorkflowPage.OnLoad(EventArgs e)     at System.Web.UI.Control.LoadRecursive()     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStag...e77f869d-19c6-0030-7223-53a406bb4c36
...esBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.HandleError(Exception e)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)    at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)     at System.Web.UI.Page.ProcessRequest()     at System.Web.UI.Page.ProcessRequest(HttpContext context)    at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)e77f869d-19c6-0030-7223-53a406bb4c36
e77f869d-19c6-0030-7223-53a406bb4c36

If someone can point me in the right direction, I'd appreciate it.

Thanks in advanced.

Fausto Capellan, Jr - SharePoint Admin

Below is the complete Error Message getting on both SharePoint 2016 and SharePoint 2013 environments:

System.InvalidOperationException: Operation failed with error Microsoft.Workflow.Client.AuthenticationException: Authentication Failed. Valid credentials must be provided for one of the following protocols: Bearer, Negotiate. HTTP headers received from the server - ActivityId: fd3b81b8-87b7-47dd-b882-cae2bdd5cf11. NodeId: ******. Scope: /SharePoint/default/e4585b78-7954-415a-9586-6084f6756411. WWW-Authenticate: Bearer realm="736a5ca1-92b1-48d0-bbdd-3550c6fac754", client_id="00000005-0000-0

Workflow manager is configured on server and able to browser the Workflow manager Site as well. All the services are running as expected. But whenever I try to publish a workflow from SharePoint Designer, getting error with Above message. Request help on this.

Hi,

I've just installed Workflow Manager 1.0 on our Dev WFE SharePoint 2013 server, following instructions from TechNet and blogs. Everything went well, registered site collection with workflow farm, verified the Workflow Service Application Proxy is in connected status and the 2013 Workflow choice shows up in the dropdown list when creating a new workflow in SP Designer. We have one App server, one WFE server in the DEV farm by the way and I'm just using the WFE server for workflow farm. However, after we make a workflow on a list and when clicking the ellipses on the workflow item and clicking workflows on the dropdown to open workflows on the item, it gives the following error.

I've searched online for solutions and was not able to find any. If you have any suggestions or ideas on solving this, it would be greatly appreciated! Thanks!

Hi

i reading any answer about this error.but

 My problem was not resolved.

when run this:

Register-SPWorkflowService -SPSite "https://server9/sites/rd" -WorkflowHostUri "https://server10:12290"

error:The Root of the certificate chain is not the trusted root authority apear

i reading and running this answer:

https://social.msdn.microsoft.com/Forums/office/en-US/026e14fa-8e22-48db-88c3-f3a5837db1b4/the-root-of-the-certificate-chain-is-not-a-trusted-root-authory-register-spworkflowservice-with?forum=sharepointcustomization

but when browse:https://server10:12290...certificate error apear

please help me.

Hi

I am attempted to setup workflow manager on an "all in one" test box where SharePoint 2013 and Workflow Manager are hosted on the same box.

Reigister-SPWorkflowSerive - SPSite "Https://intranet.domain.com" -workflowhostUri "https://sp2013allin:12290"

Causes the above error. .

I am thinking I may need to re-install the workflow manager as I think I might of replaced the personal cert after I first installed it.

Daniel

Freelance consultant

#1. How to install SharePoint 2013 Designer Workflows for SPD for SharePoint Server 2016. Kindly provide the complete steps to install and configure.

#2. I am having 3 Servers : i) App Server ii) Web Front Server iii) DB Server. In which Server we need to install Workflow Manager Server.

Tagore

#1. How to have SharePoint 2013 Designer Workflow platform for SharePoint Server 2016. Kindly provide the complete steps to install and configure.

#2. I am having 3 Servers : i) App Server ii) Web Front Server iii) DB Server. In which Server we need to install Workflow Manager Server. 

Tagore

Hi,

I have a weird problem. In a 2013 workflow, I send a mail after the approval task completes. For some random instances, this same mail is sent multiple times, some 30 times roughly counted. I can see this in the SharePoint ULS logs.

I had this same issue three times at random times during the day in the last two weeks. And in those two weeks I have at least 100 approval flows where I did not have this problem, so there is nothing wrong with the flow ;-)

Any idea where to start searching ?

Marc

Hello,

I Was using these as a Starting point for migration Workflow Manager:

But now I'm stuck with the error below

HI Team,

Please share me the steps or power shell script to generate SharePoint 2016 workflow report for particular library and site

Thanks in advance

Regards

Shyam.CH

Hello, I have the problem when trying to download WorkFlow Manager offline, to install it.

C:\Windows\system32>webpicmd /offline /Products:WorkflowManagerRefresh /Path:C:\
WorkFlow

The software that you obtain using the Web Platform Installer Command Line Tool
is licensed to you by its owner.  Microsoft grants you no rights for third party
 software.
Loading products in online feeds ...

Loading products in offline feeds...
Creating offline cache for following products:
WorkflowManagerRefresh
NETFramework452
StaticContent
WASProcessModel
DefaultDocument
DirectoryBrowse
HTTPErrors
HTTPLogging
LoggingTools
RequestMonitor
RequestFiltering
StaticContentCompression
IISManagementConsole
WASConfigurationAPI
NetFx4Extended-ASPNET45
ASPNET
NETExtensibility
WASNetFxEnvironment
NetFx3
ISAPIExtensions
ISAPIFilters
IIS7
ASPNET_REGIIS
NETFramework45
NETFramework4
WindowsImagingComponent
WindowsInstaller31
IIS51
IIS60
ASPNET45
NetFxExtensibility45
ManagementService
WindowsAuthentication
WorkflowClient
ServiceBusAnyVersion
ServiceBus_1_1
WindowsFabric
VC11Redist_x64
WindowsFabric_1_0_960_0
WindowsFabricV1RTM_1_0_944_0
NETFramework4PU3
WindowsManagementFramework
ServiceBus_1_1_NETFramework46_Update
NonSBPatch

Unexpected exception occurred
Object reference not set to an instance of an object.

i  have designed a SPD 2013 workflow. where i am reading list item by call HTTP web service.  i am getting System.ApplicationException: HTTP 500 sharepoint 2013 designererror.

though i am able use Call HTTP web service to other service like http://services.odata.org/Northwind/Northwind.svc/Customers.

Note : i am getting Request Status as OK

Error details:

RequestorId: e29dc009-ac32-c732-eccd-91a01a59eb4f. Details: System.ApplicationException: HTTP 500 {"Transfer-Encoding":["chunked"],"X-SharePointHealthScore":["0"],"SPClientServiceRequestDuration":["36"],"SPRequestGuid":["e29dc009-ac32-c732-eccd-91a01a59eb4f"],"request-id":["e29dc009-ac32-c732-eccd-91a01a59eb4f"],"X-FRAME-OPTIONS":["SAMEORIGIN"],"MicrosoftSharePointTeamServices":["15.0.0.4420"],"X-Content-Type-Options":["nosniff"],"X-MS-InvokeApp":["1; RequireReadOnly"],"Cache-Control":["max-age=0, private"],"Date":["Fri, 24 Oct 2014 14:31:43 GMT"],"Server":["Microsoft-IIS\/8.0"],"X-AspNet-Version":["4.0.30319"],"X-Powered-By":["ASP.NET"]} at Microsoft.Activities.Hosting.Runtime.Subroutine.SubroutineChild.Execute(CodeActivityContext context) at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation) 

Hi Everyone,

I'm new at this, and keep hitting a wall at ever turn.

Now I'm trying to configure Workflow Manager that I installed with Web Platform Installer.

I installed Web Platform Installer and Workflow Manager on my laptop. Is this correct?

Should I have installed both these programs on the SharePoint Server?

Is this why I get these errors?

Thanks in advance.

We required to update TLS 1.0 to TLS 1.2 in SharePoint 2013 on premises environment with server running on windows server 2008 R2.

We have workflow manager 1.0 and service bus 1.1 installed and communication between WFM and SharePoint is enabled through SSL .

For updating to TLS 1.2, we are supposed to update service bus 1.1 with TLS 1.2 along with WFM 1.0 with CU5. 

Will Windows server 2008 R2 supports service bus 1.1 with TLS 1.2 update and any Microsoft official support link on this one?

I found this below one in web platform installer .

I have a Workflow Manager and Service Bus farm configured and working. I am trying to add a second server to the farm. During configuration, as the second host is being added through Add-SBHost, there is an attempt made to stop and start Service Bus services on the first host. The start fails because the Service Bus Message Broker service gets stuck in the "Starting" state. It retries this 3 times, each try taking around 7 minutes and each time the failure ending with a message that looks like this:

ServiceBus Broker service failed to start, retry count 1. Exception message: Operation timed out..  Stack Trace:   at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()

  at Microsoft.Cloud.HostingModel.ComponentHost.OpenComponent(IComponent component, RequestTracker tracker)

  at Microsoft.Cloud.HostingModel.ComponentHost.Open()

  at Microsoft.ServiceBus.MessageBroker.Backend.OnStart(String[] args)

This error is preceded by 2 others that present no more information than the above does:

TrackingId: f8ff41e9-53e0-48fc-b4de-b7505d4d5f7b, SubsystemId: NoSystemTracker. Failed to open Messaging Host Component System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF

  at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)

  at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)

  at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)

  --- End of inner exception stack trace ---

  at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()

and...

Windows Fabric Runtime create failed Exception System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF

  at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)

  at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)

  at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)

  --- End of inner exception stack trace ---

  at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()

The Windows Fabric Runtime creation itself seems to be retried 6 times, each try taking around 70 seconds and each time it fails, the following information message is posted to the log:

Windows Fabric Runtime create failed (Retries left 6) Exception System.TimeoutException: Operation timed out. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BFF

  at System.Fabric.Interop.NativeRuntime.FabricEndCreateRuntime(IFabricAsyncOperationContext context)

  at System.Fabric.FabricRuntime.NativeFabricRuntimeFactory.InitializeFabricRuntimeEndWrapper(FabricRuntime runtime, AsyncCallOutAdapter adapter)

  at System.Fabric.Interop.Utility.FinishNativeAsyncInvoke[TResult](String functionTag, Func`2 endFunc, TaskCompletionSource`1 source, AsyncCallOutAdapter adapter, Boolean expectCompletedSynchronously)

  --- End of inner exception stack trace ---

  at System.Fabric.FabricRuntime.Create(Action fabricExitCallback)

  at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()

Can anyone throw some light on what could be causing this failure? I am hoping the answer is not uninstall everything and reinstall.

Hi All. Can you please help me, i was tasked to create a list where in employees can put their request for leave.

Now, my problem are:

1. How can I can create a workflow where approver1 has alternate approver specially when approver1 is not around. We have 3 approvers and all of them needs to have alternate approver. If the main approvers is not around the alternate approver will be receiving an email for the said request and needs to be approved/reject/task assigned.

2. How to create workflow with task assigned like the normal workflow in SharePoint.

I will be using SharePoint Designer 2013 in creating this Workflow Approval

I hope you understood what is my help need.

Thank you so much,

Karen

Hi All, 

Service Bus Message Broker always stuck at starting/pending.

Please help

Guys,

I still can't get Workflow 2013 to properly process workflow on my servers. I read articles about permissions and other ways to get the User Profile Service to run. Below are my questions.

I have 4 web servers and 2 SQL server. I installed workflow manager on 2 servers but in reality, I just need Workflow on 1 server. My question, does the User Synchronization Service need to be running on all servers? Or, does it need to just be running on the server where Workflow was configured? 

Microsoft manager to programmer: You start coding. I'll go find out what they want ...

As described in topicname the Service Bus Message Broker stucks at starting with error below.
I followed all post on technet, blogs etc. restarting services, restart service, hosts file, no proxy, no net 4.6 is installed but all no success and services still stuck starting.

Windows Fabric service registration failed. Exception System.Fabric.FabricException: Service type is already registered. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BD1
   at System.Fabric.Interop.NativeRuntime.IFabricRuntime.RegisterStatefulServiceFactory(IntPtr serviceType, IFabricStatefulServiceFactory factory)
   at System.Fabric.FabricRuntime.InternalRegisterServiceFactory(Boolean isStateful, String serviceType, ServiceFactoryBroker serviceFactoryBroker)
   at System.Fabric.FabricRuntime.RegisterServiceTypeHelper(Boolean isStateful, String serviceTypeName, Type serviceTypeImplementation)
   at System.Fabric.Interop.Utility.<>c__DisplayClassad.<WrapNativeSyncInvoke>b__ac()
   at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag)
   --- End of inner exception stack trace ---
   at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag)
   at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()

TrackingId: a3d58b4c-5363-4952-a500-cc7a6c55ff56_BAPP28, SubsystemId: NoSystemTracker. Failed to open Messaging Host Component System.Fabric.FabricException: Service type is already registered. ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80071BD1
   at System.Fabric.Interop.NativeRuntime.IFabricRuntime.RegisterStatefulServiceFactory(IntPtr serviceType, IFabricStatefulServiceFactory factory)
   at System.Fabric.FabricRuntime.InternalRegisterServiceFactory(Boolean isStateful, String serviceType, ServiceFactoryBroker serviceFactoryBroker)
   at System.Fabric.FabricRuntime.RegisterServiceTypeHelper(Boolean isStateful, String serviceTypeName, Type serviceTypeImplementation)
   at System.Fabric.Interop.Utility.<>c__DisplayClassad.<WrapNativeSyncInvoke>b__ac()
   at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag)
   --- End of inner exception stack trace ---
   at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag)
   at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
   at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()

ServiceBus Broker service failed to start, retry count 3.  Exception message: Service type is already registered..  Stack Trace:    at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag)
   at Microsoft.Cloud.ServiceBus.MessageContainerHost.Ring.Join()
   at Microsoft.Cloud.ServiceBus.MessageContainerHost.MessageContainerHostComponent.Open()
   at Microsoft.Cloud.HostingModel.ComponentHost.OpenComponent(IComponent component, RequestTracker tracker)
   at Microsoft.Cloud.HostingModel.ComponentHost.Open()
   at Microsoft.ServiceBus.MessageBroker.Backend.OnStart(String[] args)

and further in application log

Application: Microsoft.ServiceBus.MessageBroker.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: Failed to start Service Bus Broker Service.
Stack:
   at System.Environment.FailFast(System.String, System.Exception)
   at Microsoft.ServiceBus.MessageBroker.Backend.OnStart(System.String[])
   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Faulting application name: Microsoft.ServiceBus.MessageBroker.exe, version: 2.0.20922.0, time stamp: 0x505e1bad
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0x80131623
Fault offset: 0x00007fff32b90392
Faulting process id: 0x19a0
Faulting application start time: 0x01d16e5c9c05e329
Faulting application path: C:\Program Files\Service Bus\1.0\Microsoft.ServiceBus.MessageBroker.exe
Faulting module path: unknown
Report Id: 70ee2d3b-da52-11e5-80c8-005056bc5c8a
Faulting package full name: 
Faulting package-relative application ID: 

get-sbfarm

FarmType                      : SB
SBFarmDBConnectionString      : Data Source=workflow;Initial Catalog=SbManageme
                                ntDB;Integrated Security=True;Encrypt=False
ClusterConnectionEndpointPort : 9000
ClientConnectionEndpointPort  : 9001
LeaseDriverEndpointPort       : 9002
ServiceConnectionEndpointPort : 9003
RunAsAccount                  : contoso\sa-wflow
AdminGroup                    : contoso\SG wflowadmin
GatewayDBConnectionString     : Data Source=workflow;Initial Catalog=SbGatewayD
                                atabase;Integrated Security=True;Encrypt=False
HttpsPort                     : 9355
TcpPort                       : 9354
MessageBrokerPort             : 9356
FarmCertificate               : Thumbprint: 7B64670F9A39D011B32C9C23245438EB836
                                CE850, IsGenerated: True
EncryptionCertificate         : Thumbprint: 7B64670F9A39D011B32C9C23245438EB836
                                CE850, IsGenerated: True
Hosts                         : {Name: WF01.network.contoso.com, Configuration Sta
                                te: HostConfigurationCompleted}

get-sbfarmstatus

             HostId HostName            ServiceName                      Status
             ------ --------            -----------                      ------
                  1 wf01.network.co... Service Bus Gateway             Running
                  1 wf01.network.co... Service Bus Mess...        StartPending
                  1 wf01.network.co... FabricHostSvc                   Running

Single server and all self signed certificates. Certificates all trusted and in trusted root ca auth store in MMC certificate snapin.
Port 9000 is not in use by other service, when i kill service bus message broker and run stop-sbfarm and then run netstat -ano i cannot find a process that listens on port 9000 (0.0.0.0:9000) so its not in use. Then Fabric service starts and is running but service bus message broker hangs on starting.