Add dual TLS/Plaintext networking support

[Tim-Brooks]

Currently we do not offer a straightforward way to enable security on a cluster without doing a full cluster restart. This is because a cluster cannot communicate when half of it is plaintext and half of it is TLS.

We can support both TLS and plaintext communications on new nodes that have enabled security. This would allow a rolling restart when enabling security.

There solution is:

1. On incoming connections, check if the first incoming bytes are 'E', 'S'. These bytes do not overlap with TLS bytes. If the first bytes are 'E', 'S' then it is a plaintext connection. Otherwise assume it is TLS.
2. On outgoing connections, attempt to open a TLS connection. If the other node is plaintext that connection will fail quickly. If the connection fails, attempt a plaintext connection.

There are two tricky parts of this issue:

1. Normally we do not want to allow dual stack. @jaymode and I discussed a number of potential solutions and we were struggling to find a good one. We could add some type of dynamic setting that indicates if dual stack is supported. However, there are a number of failure scenarios here as a restarted node with security NEEDS to know if it supports plaintext before joining a cluster. To ensure that the restarted node knows this, the customer would need to add this setting to each of the nodes yml files. And then they have to update the setting (once the rolling restart is complete) to disable this functionality.

2. We also need to have some type of reaper process that will kill plaintext connections once the dual stack is disabled. This is problematic as it is possible that a connection is open, but we have not yet identified if it is plaintext. I think in this scenario we just want to kill the connection anyway. We could also check on every channel read/write on a dual stack plaintext channel if the dual stack networking is still enabled.

I think we need to discuss the best approach to the first issue. I can probably resolve the second one once we know how we are approaching the enablement of this functionality.

@s1monw @jaymode

[elasticmachine]

Pinging @elastic/es-security

[DaveCTurner]

However, there are a number of failure scenarios here as a restarted node with security NEEDS to know if it supports plaintext before joining a cluster.

Could you describe these scenarios in more detail?

Also wanted to note that we're already not 100% failure tolerant during rolling upgrades. For instance if you get halfway through a rolling upgrade from 5.6 to 6.x and then all the remaining 5.6 nodes are temporarily disconnected from the 6.x master then they may not be allowed to rejoin the cluster because of MembershipAction#ensureMajorVersionBarrier(). Maybe it's ok to be similarly intolerant of failures when enabling security in a rolling fashion?

[Tim-Brooks]

Could you describe these scenarios in more detail?

I mean lets say that dual_stack: enabled is a persistent dynamic setting and it is false by default. You would need to apply that setting before doing your rolling restart. The failure scenario is if, for any reason, a node did not receive that setting before restarting, it would not be able to join the cluster after restart (as it would only be speaking TLS and the cluster would be speaking plaintext).

Additionally, if you wanted to add nodes during the rolling restart, you would need to apply the setting locally.

I'm not saying that scenario (a node not receiving a dynamic setting) is likely. It just seems to introduce confusing failure scenarios. Obviously this can be avoided by applying the setting to every yml file. But that is some overhead to users.

And I wanted to make sure we had an agreement on how we wanted to handle the "indicator" that we should accept plaintext connections. If a setting is the correct approach?

[s1monw]

thanks @tbrooks8 for starting this initiative. The way I was thinking about this was more of an anti-viral setting or a one-way door, once it's closed you can't open it anymore. Lemme explain, the typical scenario here is you have an existing cluster that is plain text and you want to upgrade. New nodes will have a dual stack and old nodes won't. Once you join a cluster with a new node we start to connect to all nodes in the cluster. Once we are connected to all nodes in the cluster ie. fully joined we switch the plain text transport into it's anti-viral mode which means once it's last connection is closed it won't accept any new connections anymore and will never try to use plain text. This is typically the moment when the last node that uses plaintext is shut-down. This also means that if you restart a node after we crossed that point can safely go through the same process again and will end up automatically disabling itself once it has fully joined the cluster. The cluster will then not allow nodes with plain text to join anymore.

does this make sense or do I miss anything?

[Tim-Brooks]

This is typically the moment when the last node that uses plaintext is shut-down.

Do you have a thought about what our indicator is here? Like when we start a new node and always accept TLS and plaintext. And then have a listener for when a node disconnects, we check and see if all the nodes a now TLS. If they are all TLS, we stop accepting new plaintext connects?

Or is there somewhere else the listener should be attached?

Does this make sense from a security perspective @jaymode?

[s1monw]

Do you have a thought about what our indicator is here? Like when we start a new node and always accept TLS and plaintext. And then have a listener for when a node disconnects, we check and see if all the nodes a now TLS. If they are all TLS, we stop accepting new plaintext connects?

yeah that is what I was thinking about.

[jaymode]

Does this make sense from a security perspective

It does; that said I want to lay out what could be seen as a "weakness" here in that during a rolling restart, nodes would allow plaintext initially and with a hijack of DNS someone could essentially have the cluster join with nodes it shouldn't join with and then gain access to data. This would be a pretty tough thing to do and at that point you probably have much worse things to worry about if part of your critical infrastructure has been comprised.

Do we plan on having a setting to disable this behavior at the node level? Also, cc @elastic/es-security to keep me honest.

[s1monw]

+1 or have a setting