The connection of Integration Runtime instances cannot always recover when connection lost in ThingWorx HA

• In ThingWorx HA, integration runtime instances are connected to connection servers through load balancer Nginx
• There are 2 or more connection servers up and running in ThingWorx HA
• Using QueryEndpointSessions and GetBoundThingsForEndpoint services of WSCommunication Subsystem to check which connection server IR is connected with
• Manually terminate that connection server to shut it down
• The connected Integration Runtime is expected to recover from connection failure by routing the connection to other available connection servers with the help of load balancer
• IR log stuck at below error message and not showing any sign of recover attempt

[Timer-0] ERROR c.t.i.u.RemoteServiceUtils - Error while invoking Health Check on Thingworx
java.lang.Exception: No open connections were available on endpoint 0
	at com.thingworx.communications.common.endpoints.CommunicationEndpoint.getConnection(CommunicationEndpoint.java:397)
	at com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendMessage(CommunicationEndpoint.java:416)
	at com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendRequest(CommunicationEndpoint.java:531)
	at com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendRequest(CommunicationEndpoint.java:500)
	at com.thingworx.communications.client.BaseClient.sendRequest(BaseClient.java:1076)
	at com.thingworx.communications.client.BaseClient.invokeService(BaseClient.java:418)
	at com.thingworx.communications.client.BaseClient.invokeService(BaseClient.java:372)
	at com.twx.integration.client.things.IntegrationRuntimeServer.invokeService(IntegrationRuntimeServer.java:101)
	at com.twx.integration.utils.RemoteServiceUtils.getHealthResponse(RemoteServiceUtils.java:156)
	at com.twx.integration.Entrypoint$HealthTask.run(Entrypoint.java:112)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)
[Timer-0] WARN  c.t.c.c.BaseClient - Unable to set message authentication because Client CommunicationEndpoint is not authenticated
[Client-EndpointMonitor-1] ERROR c.t.c.c.ConnectedThingClient - Unable To Bind Things: Could not bind things: Error trying to process message: Binding failed, see server log for more information V1BindMessage [header=V1MessageHeader [messageType=10, requestId=101, endpointId=1, sessionId=1670095344, flags=0], gatewayName=IR-WIN-QGPP8TIB5JV-ef087aef-ef7c-44a1-92b1-f0c77bbaecad, gatewayType=IntegrationRuntime, names=[IR-WIN-QGPP8TIB5JV-ef087aef-ef7c-44a1-92b1-f0c77bbaecad, IR-WIN-QGPP8TIB5JV-ef087aef-ef7c-44a1-92b1-f0c77bbaecad]]

• Sometime reconnect attempt do occur and can find below log in IR log

[Client-EndpointMonitor-1] ERROR c.t.c.c.e.ClientCommunicationEndpoint - Refilling connections on [endpoint 0, uri: ws://<ip>:<port>/Thingworx/WS] failed : timeout waiting for websocket handshake to complete
[Client-EndpointMonitor-1] ERROR c.t.c.c.e.ClientCommunicationEndpoint - Exception refilling connections was:
java.util.concurrent.TimeoutException: timeout waiting for websocket handshake to complete
	at com.thingworx.communications.client.connection.netty.NettyClientConnectionFactory$HandshakeCompletionNotifier.await(NettyClientConnectionFactory.java:693)
	at com.thingworx.communications.client.connection.netty.NettyClientConnectionFactory.createConnection(NettyClientConnectionFactory.java:398)
	at com.thingworx.communications.client.endpoints.ClientCommunicationEndpoint.refillConnections(ClientCommunicationEndpoint.java:230)
	at com.thingworx.communications.common.endpoints.monitor.ConnectivityMonitorTask.dutyCycleOn(ConnectivityMonitorTask.java:159)
	at com.thingworx.communications.common.endpoints.monitor.ConnectivityMonitorTask.run(ConnectivityMonitorTask.java:306)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)