Add Kerberos Realm

[elasticmachine]

Original comment by @skearns64:

Kerberos is an increasingly common auth system and is a commonly requested integration for Shield. With pluggable realms, a user could create their own realm that integrates with Kerberos, but having a built-in realm that integrates with Kerberos would make it significantly easier to integrate into these environments.

We should consider making this available.

[elasticmachine]

Original comment by @skearns64:

Note that this realm would support Kerberos over HTTP, and would also support SPNEGO.

[elasticmachine]

Original comment by @clintongormley:

Well worth a read: LINK REDACTED

[elasticmachine]

Original comment by @pmoust:

FYI: Cloud expresses interest in Kerberos /cc @MikePaquette @joshbressers

[elasticmachine]

Original comment by @jaymode:

Can you explain the use case for cloud?

[elasticmachine]

Original comment by @jaymode:

@pmoust Can you please add more details about Cloud's use case for kerberos? Without that we cannot be sure that what we develop will meet the needs of cloud.

[elasticmachine]

Original comment by @jaymode:

Kerberos support will be implemented via HTTP SPNEGO. HTTP SPNEGO can be implemented in Java using the Java GSS api.

The typical flow for HTTP SPNEGO is:

1. User makes request
2. Server responds with HTTP 401 with the WWW-Authenticate: Negotiate header
3. Client determines whether it should negotiate with the server; Internet Explorer will only Negotiate with sites that are in the local intranet or trusted sites zone. For firefox, the configuration parameter network.negotiate-auth.trusted-uris needs to contain the uri
4. Client sends request again with header Authorization: Negotiate ...
5. Server validates the value of the Negotiate header using a Keytab file

A few important items to note, the Negotiate protocol can return NTLM tokens instead of kerberos. We should not attempt to provide support for NTLM tokens. There may be more than one roundtrip needed to authenticate the kerberos token and/or the token can be authenticated but data needs to be sent back to the client.

If we think about Kibana users wanting to authenticate via HTTP SPNEGO (common in Microsoft environments for SSO) then Kibana will be the interaction point with the user. This means that the browser will request a token to be sent to the service principal for Kibana. This means that Elasticsearch will need to be aware of the service principal name for Kibana. In terms of the Kibana flow, I believe that it should be somewhat similar to the SAML flow:

1. User accesses Kibana
2. User is challenged - TBD if this is from within Kibana or Elasticsearch
3. User presents token; Kibana passes it to Elasticsearch
4. If necessary, repeat challenge
5. Kibana retrieves token from Elasticsearch for user - TBD will this be like the SAML API or should Kibana handle the decoding of the WWW-Authenticate response?

Other HTTP clients should also be able to perform HTTP SPNEGO. This will be necessary for integration with ES Hadoop and possibly logstash/beats down the line. This means that Elasticsearch will also need to know about its own service principal name.

The next aspect of a Kerberos realm is how we authorize the principal after authentication. The typical Kerberos ticket only provides a subject, so we should be able to query an LDAP instance for additional group information and perform role mapping based upon the principal.

It should be considered out of scope to implement special handling of PAC data in Microsoft's tickets for the first iteration. For completeness, there is an unmaintained library that decodes this data , jaaslounge. The latest release of the library does not support AES encrypted tokens, but the code can easily be modified to support this.

@kobelb @legrego @elastic/es-security thoughts?

[elasticmachine]

Original comment by @legrego:

@jaymode This approach makes sense to me for SPNEGO-enabled clients. Will we be able to do anything for clients that don’t support SPNEGO?

Regarding the Kibana flow:

2. User is challenged - TBD is this is from within Kibana or Elasticsearch

I think Kibana can handle the initial challenge if no authorization header is present (via www-authenticate: negotiate). Any repeat challenges (step 4 above) could come from Elasticsearch

5. Kibana retrieves token from Elasticsearch for user - TBD will this be like the SAML API or should Kibana handle the decoding of the WWW-Authenticate response?

My initial thought here is to have Elasticsearch perform the decoding, so that we're consistent with the SAML implementation, and so other clients won't also have to handle this. This part of the auth flow is a little unclear to me though, so what are the tradeoffs of each approach?

[elasticmachine]

Original comment by @jaymode:

Will we be able to do anything for clients that don’t support SPNEGO?

Can you clarify what you mean by clients that do not support SPNEGO? I do not think we would implement anything that would allow them to use Kerberos if they do not support SPNEGO.

I think Kibana can handle the initial challenge if no authorization header is present (via www-authenticate: negotiate). Any repeat challenges (step 4 above) could come from Elasticsearch

I personally like this for Kibana. There are a few reasons; this would allow Kibana to challenge for negotiate authentication but still fallback to username/password for clients that do not support negotiate auth. I've done this in the past by doing the following: return a HTTP 401 with the www-authenticate: negotiate header AND a html body:

<html>
    <head>
        <meta http-equiv="refresh" content="0;URL='LINK REDACTED" />    
    </head>
</html>

A browser that does not support negotiate for a given site, will not provide any response to the server so the body trick allows you to redirect the user to the login page. If the browser support negotiate, the body gets ignored.

My initial thought here is to have Elasticsearch perform the decoding, so that we're consistent with the SAML implementation, and so other clients won't also have to handle this. This part of the auth flow is a little unclear to me though, so what are the tradeoffs of each approach?

I realize how unclear my statement was there. The two options would be:

1. Have a special endpoint for Kibana to post requests with Kerberos tokens, like we have for SAML. This special API in elasticsearch that Kibana posts the SAML response to, returns a response that contains the token.

2. Kibana checks for a user having a token/auth and if none starts the SPNEGO flow and proxies the requests to elasticsearch. The request would be different than the users request and would be for a token (ie hit the POST /_xpack/security/oauth2/token endpoint) and the response there will return a token that can be used for continued authentication of the user. special API was created in elasticsearch that Kibana would post the SAML response to and the response would contain the token.

The more I think about it, I feel like we do not need to add a special endpoint.

[elasticmachine]

Original comment by @legrego:

Can you clarify what you mean by clients that do not support SPNEGO? I do not think we would implement anything that would allow them to use Kerberos if they do not support SPNEGO.

You actually answered my question in your followup, about returning an HTML response alongside the WWW-Authenticate challenge. In this scenario, Kibana would initiate the standard username/password authentication request to Elasticsearch, and then authentication would be attempted against the available realms in the chain capable of fulfilling the request?

2. Kibana checks for a user having a token/auth and if none starts the SPNEGO flow and proxies the requests to elasticsearch. The request would be different than the users request and would be for a token (ie hit the POST /_xpack/security/oauth2/token endpoint) and the response there will return a token that can be used for continued authentication of the user. special API was created in elasticsearch that Kibana would post the SAML response to and the response would contain the token.

This approach makes sense to me. We can extract the token from the authenticate header and pass that along to the existing Token Service endpoint above. Elasticsearch will then exchange that for an access token after performing authentication.

[elasticmachine]

Original comment by @pmoust:

Thanks for pinging @clintongormley and @jaymode - sorry for taking us too long to reply.
/cc @robgil
We'll get back to you within this week.

[elasticmachine]

Original comment by @robgil:

In cloud we will be running our own Directory Services (FreeIPA). This supports Kerberos, LDAP, HBAC, RBAC, central sudo, among others. We can certainly just go with SAML to integrate with both kerberos and LDAP, but direct kerberos would be handy as well. This should be a simple as adding a kerberos service principal for each ECE service. So for each cluster, Kibana and ES would each get their own service principals. This would allow us to grant access to each service independently and manage within FreeIPA. Granted kerberos is only AuthZ, AuthN would need to be handled elsewhere regardless, whether it be within ECE, or privilege/group mappings in LDAP.

This use case would essentially be the same for everyone. I don't think we should run our own Kerberos realm within ECE. Kerberos realms/masters are typically run by the organization already, and there is no need to add an additional master. Kerberos can be integrated with AD, FreeIPA, Fedora Directory Services, and others.

Direct built in kerberos would also be very handy for securing ES access. Since applications are typically not user accounts, using kerberos over http for ES would allow role account access to ES without getting redirected to SAML for auth.

[elasticmachine]

Original comment by @jaymode:

Thanks for the details @robgil @pmoust. This is important especially for the Kibana team as there will be additional work there for Kibana server to Elasticsearch authentication using Kerberos, cc @legrego. I have no idea what the kerberos support is like in node.js, so I would treat this as a separate item that end user kerberos support for Kibana.

[elasticmachine]

Original comment by @legrego:

As @jaymode mentioned in the cross sync meeting today, node.js doesn't have built-in kerberos support.
There is an Apache-2.0 licensed LINK REDACTED, which is written as a LINK REDACTED. Using this requires us to build platform specific binaries for each of our supported environments, which is similar to what @kobelb is doing for his syscall filtering work.

It sounds like the Cloud team won't need this support right away, so this could potentially be worked on in a later phase