Idle elastic java low level client continuously create new TCP socks

[fangqiao]

I was testing a new backend service, which created connections to a cluster of three boxes but it didn't use the connections at all. I accidentally found out it was creating 3 new tcp socks every second, when I ran lsof -p pid to examine open file usage.

I commented out all communication modules, with only few lines of code creating es connections left. It became obvious that es connection caused the problem. My production system didn't have the problem as it actively used the es connections.

I guess it is the es connection heart beat that continuously opening new TCP socks every second but it didn't close them.

I think this is an issue that should be fixed. It likely opens too many files that will cause serious problems.

I am running es 5.6.0 server and java client.

[ywelsch]

Is this the REST client or the transport client? Can you show us a code excerpt how you create said client?

[javanna]

@fangqiao can you please post your client code?

[fangqiao]

It is transport client. I use scala 2.12. That's all the code left. Then I ran lsof -p pid | grep TCP | wc -l every second. The count increased by 3 exactly the same number of es servers configured for the connection.

// BjConfig is configuration object.
    val esServers = BjConfig.EsHosts.zip(BjConfig.EsPorts) // Build host and port pair.

    val esSettings = Settings.builder().
      put("cluster.name", BjConfig.EsClusterName).
      put("client.transport.sniff", true).
      build()

    val esClient: TransportClient = new PreBuiltTransportClient(esSettings) 

    esServers.foreach(es => {
                        esClient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(es._1), es._2.toInt))
                      })

[javanna]

Hi @fangqiao ,
yes the connections are due to sampling, in your case sniffing as you enabled sniffing in your configuration. We should be closing connections though...it would be pretty bad if we didn't.

@bleskes I see that you made some changes a while ago on sampling and connections handling, as part of #22828 . Do you have thoughts on this?

[rjernst]

This issue is quite old, and we have not heard of other reports of sampling leaving idle connections that I know of. As the transport client is deprecated, I'm going to close this issue. If any user can reproduce this issue on a current version (7.x) we can investigate further and fix (regardless of the transport client's deprecation).