Regression in SPN generation in .NET 5.0 for non-standard HTTP ports

[mattpwhite]

It appears that SocketsHttpHandler was changed in .NET 5.0 (#40860) to unconditionally include the port in the Kerberos service principal name when the HTTP service runs on a non-default port. The intent of that change appears to have been to emulate the behavior of .NET Framework, but it actually created a divergence from .NET Framework behavior, at least for environments that expect out-of-the-box default behavior (no port in SPN).

The inclusion of the port in the SPN is a thing IE could be configured to do (the original KB is lost to time, but it's referenced many places, e.g. https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/internet-explorer-behaviors-with-kerberos-authentication/ba-p/396428). To the best of my knowledge, no browser includes the port by default. Chrome has an option (again, off-by-default) to include the port - https://www.chromium.org/developers/design-documents/http-authentication (see "Kerberos SPN Generation"). Perhaps there was also something that could be done to configure .NET Framework to do this also, but again, it is not the default.

My guess is that this change broke fewer things than one might expect because, in a pure Windows shop, assuming security hasn't been tightened, you get an auto-magic downgrade to NTLM after the KDC returns an error because a principal with a port number in it does not exist. If NTLM is unsupported for one reason or another (Linux stuff in the mix, you care about security and have disabled NTLM) the fallback won't happen and things just don't work.

Given the following test program, I observe consistent behavior in .NET 4.8 and .NET Core 3.1, but .NET 5.0 diverges. My test below only has Windows output, but this regression is also observed in Linux .NET Core clients. This is blocking our ability to port several legacy codebases to .NET Core or upgrade from .NET Core 3.1.

Client code:

using System;
using System.Diagnostics;
using System.Net.Http;
using System.Runtime.InteropServices;
using System.Threading.Tasks;

namespace SpnPortDemo
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine($"Running under: {RuntimeInformation.FrameworkDescription}");
            RunConsoleApp("klist", "purge");

            try
            {
                var client = new HttpClient(new HttpClientHandler {UseDefaultCredentials = true});
                Console.WriteLine("Sending HTTP GET to " + args[0]);
                using (var response = await client.GetAsync(args[0]))
                {
                    Console.WriteLine($"Status: {response.StatusCode}");
                    Console.WriteLine("Body:");
                    Console.WriteLine(await response.Content.ReadAsStringAsync());
                }
            }
            catch (HttpRequestException e)
            {
                Console.WriteLine(e);
            }

            RunConsoleApp("klist");
        }

        private static void RunConsoleApp(string file, string args = null)
        {
            var psi = new ProcessStartInfo
            {
                FileName = file,
                Arguments = args,
                CreateNoWindow = true,
                UseShellExecute = false,
                RedirectStandardOutput = true,
            };

            Console.WriteLine();
            Console.WriteLine($"Output from: {file} {args}");
            var proc = Process.Start(psi);
            Console.WriteLine(proc.StandardOutput.ReadToEnd());
            Console.WriteLine();
        }
    }
}

Dummy (Windows only) server code (run as SYSTEM using psexec -sid cmd)

using System;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace ListenerTest
{
    class Program
    {
        static int Main(string[] args)
        {
            if (args.Length != 1 || !ushort.TryParse(args[0], out var port))
            {
                Console.WriteLine("Failed to parse port");
                return 1;
            }

            var listener = new HttpListener();
            listener.Prefixes.Add($"http://+:{port}/");
            listener.AuthenticationSchemes = AuthenticationSchemes.Negotiate;
            Console.WriteLine($"Starting listener on port {port}");
            listener.Start();

            Task.Run(() =>
            {
                try
                {
                    while (true)
                    {
                        var ctx = listener.GetContext();
                        var bytes = Encoding.UTF8.GetBytes($"{ctx.User.Identity.AuthenticationType}: {ctx.User.Identity.Name}");
                        ctx.Response.ContentLength64 = bytes.Length;
                        ctx.Response.OutputStream.Write(bytes, 0, bytes.Length);
                        ctx.Response.OutputStream.Close();
                    }
                }
                catch (Exception e)
                {
                    if (!listener.IsListening)
                        return;

                    Console.WriteLine($"Error: {e}");
                }
            });

            Console.ReadLine();
            return 0;
        }
    }
}

.NET 4.8 output - Kerberos works (note client and server must be separate boxes or you'll get NTLM anyway):

Running under: .NET Framework 4.8.4341.0

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
Kerberos: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (2)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 12:59:55 (local)
        End Time:   5/24/2021 22:59:55 (local)
        Renew Time: 5/24/2021 22:59:55 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc1.domain.com

#1>     Client: testuser @ DOMAIN.COM
        Server: HTTP/testhost.domain.com @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
        Start Time: 5/24/2021 12:59:55 (local)
        End Time:   5/24/2021 22:59:55 (local)
        Renew Time: 5/24/2021 22:59:55 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0
        Kdc Called: dc1.domain.com

.NET Core 3.1, again Kerberos works

Running under: .NET Core 3.1.15

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
Kerberos: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (2)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 13:01:49 (local)
        End Time:   5/24/2021 23:01:49 (local)
        Renew Time: 5/24/2021 23:01:49 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc1.domain.com

#1>     Client: testuser @ DOMAIN.COM
        Server: HTTP/testhost.domain.com @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
        Start Time: 5/24/2021 13:01:49 (local)
        End Time:   5/24/2021 23:01:49 (local)
        Renew Time: 5/24/2021 23:01:49 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0
        Kdc Called: dc1.domain.com

But in .NET 5.0, Kerberos fails. A packet capture shows the TGS-REQ for HTTP/testhost.domain.com:12345, which isn't found, which then triggers a fallback to NTLM. My dummy server/test environment here has NTLM enabled, so things "work". Our production environment does not support does not support NTLM.

Running under: .NET 5.0.6

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
NTLM: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (1)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 15:28:29 (local)
        End Time:   5/25/2021 1:28:29 (local)
        Renew Time: 5/25/2021 1:28:29 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc2.domain.com

I'm sure there are environments that do depend on the port in the SPN, so simply reverting might cause problems for them. .NET Core should be changed to either do it conditionally in the same manner as .NET Framework (assuming the framework checked/inherited behavior from some system level setting), or to expose a configurable knob for people that need it to opt in.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

It appears that SocketsHttpHandler was changed in .NET 5.0 (#40860) to unconditionally include the port in the Kerberos service principal name when the HTTP service runs on a non-default port. The intent of that change appears to have been to emulate the behavior of .NET Framework, but it actually created a divergence from .NET Framework behavior, at least for environments that expect out-of-the-box default behavior (no port in SPN).

The inclusion of the port in the SPN is a thing IE could be configured to do (the original KB is lost to time, but it's referenced many places, e.g. https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/internet-explorer-behaviors-with-kerberos-authentication/ba-p/396428). To the best of my knowledge, no browser includes the port by default. Chrome has an option (again, off-by-default) to include the port - https://www.chromium.org/developers/design-documents/http-authentication (see "Kerberos SPN Generation"). Perhaps there was also something that could be done to configure .NET Framework to do this also, but again, it is not the default.

My guess is that this change broke fewer things than one might expect because, in a pure Windows shop, assuming security hasn't been tightened, you get an auto-magic downgrade to NTLM after the KDC returns an error because a principal with a port number in it does not exist. If NTLM is unsupported for one reason or another (Linux stuff in the mix, you care about security and have disabled NTLM) the fallback won't happen and things just don't work.

Given the following test program, I observe consistent behavior in .NET 4.8 and .NET Core 3.1, but .NET 5.0 diverges. My test below only has Windows output, but this regression is also observed in Linux .NET Core clients. This is blocking our ability to port several legacy codebases to .NET Core or upgrade from .NET Core 3.1.

Client code:

using System;
using System.Diagnostics;
using System.Net.Http;
using System.Runtime.InteropServices;
using System.Threading.Tasks;

namespace SpnPortDemo
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine($"Running under: {RuntimeInformation.FrameworkDescription}");
            RunConsoleApp("klist", "purge");

            try
            {
                var client = new HttpClient(new HttpClientHandler {UseDefaultCredentials = true});
                Console.WriteLine("Sending HTTP GET to " + args[0]);
                using (var response = await client.GetAsync(args[0]))
                {
                    Console.WriteLine($"Status: {response.StatusCode}");
                    Console.WriteLine("Body:");
                    Console.WriteLine(await response.Content.ReadAsStringAsync());
                }
            }
            catch (HttpRequestException e)
            {
                Console.WriteLine(e);
            }

            RunConsoleApp("klist");
        }

        private static void RunConsoleApp(string file, string args = null)
        {
            var psi = new ProcessStartInfo
            {
                FileName = file,
                Arguments = args,
                CreateNoWindow = true,
                UseShellExecute = false,
                RedirectStandardOutput = true,
            };

            Console.WriteLine();
            Console.WriteLine($"Output from: {file} {args}");
            var proc = Process.Start(psi);
            Console.WriteLine(proc.StandardOutput.ReadToEnd());
            Console.WriteLine();
        }
    }
}

Dummy (Windows only) server code (run as SYSTEM using psexec -sid cmd)

using System;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace ListenerTest
{
    class Program
    {
        static int Main(string[] args)
        {
            if (args.Length != 1 || !ushort.TryParse(args[0], out var port))
            {
                Console.WriteLine("Failed to parse port");
                return 1;
            }

            var listener = new HttpListener();
            listener.Prefixes.Add($"http://+:{port}/");
            listener.AuthenticationSchemes = AuthenticationSchemes.Negotiate;
            Console.WriteLine($"Starting listener on port {port}");
            listener.Start();

            Task.Run(() =>
            {
                try
                {
                    while (true)
                    {
                        var ctx = listener.GetContext();
                        var bytes = Encoding.UTF8.GetBytes($"{ctx.User.Identity.AuthenticationType}: {ctx.User.Identity.Name}");
                        ctx.Response.ContentLength64 = bytes.Length;
                        ctx.Response.OutputStream.Write(bytes, 0, bytes.Length);
                        ctx.Response.OutputStream.Close();
                    }
                }
                catch (Exception e)
                {
                    if (!listener.IsListening)
                        return;

                    Console.WriteLine($"Error: {e}");
                }
            });

            Console.ReadLine();
            return 0;
        }
    }
}

.NET 4.8 output - Kerberos works (note client and server must be separate boxes or you'll get NTLM anyway):

Running under: .NET Framework 4.8.4341.0

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
Kerberos: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (2)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 12:59:55 (local)
        End Time:   5/24/2021 22:59:55 (local)
        Renew Time: 5/24/2021 22:59:55 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc1.domain.com

#1>     Client: testuser @ DOMAIN.COM
        Server: HTTP/testhost.domain.com @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
        Start Time: 5/24/2021 12:59:55 (local)
        End Time:   5/24/2021 22:59:55 (local)
        Renew Time: 5/24/2021 22:59:55 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0
        Kdc Called: dc1.domain.com

.NET Core 3.1, again Kerberos works

Running under: .NET Core 3.1.15

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
Kerberos: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (2)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 13:01:49 (local)
        End Time:   5/24/2021 23:01:49 (local)
        Renew Time: 5/24/2021 23:01:49 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc1.domain.com

#1>     Client: testuser @ DOMAIN.COM
        Server: HTTP/testhost.domain.com @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
        Start Time: 5/24/2021 13:01:49 (local)
        End Time:   5/24/2021 23:01:49 (local)
        Renew Time: 5/24/2021 23:01:49 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0
        Kdc Called: dc1.domain.com

But in .NET 5.0, Kerberos fails. A packet capture shows the TGS-REQ for HTTP/testhost.domain.com:12345, which isn't found, which then triggers a fallback to NTLM. My dummy server/test environment here has NTLM enabled, so things "work". Our production environment does not support does not support NTLM.

Running under: .NET 5.0.6

Output from: klist purge

Current LogonId is 0:0x576ffb0
        Deleting all tickets:
        Ticket(s) purged!


Sending HTTP GET to http://testhost.domain.com:12345
Status: OK
Body:
NTLM: DOMAIN\testuser

Output from: klist

Current LogonId is 0:0x576ffb0

Cached Tickets: (1)

#0>     Client: testuser @ DOMAIN.COM
        Server: krbtgt/DOMAIN.COM @ DOMAIN.COM
        KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
        Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
        Start Time: 5/24/2021 15:28:29 (local)
        End Time:   5/25/2021 1:28:29 (local)
        Renew Time: 5/25/2021 1:28:29 (local)
        Session Key Type: AES-256-CTS-HMAC-SHA1-96
        Cache Flags: 0x1 -> PRIMARY
        Kdc Called: dc2.domain.com

I'm sure there are environments that do depend on the port in the SPN, so simply reverting might cause problems for them. .NET Core should be changed to either do it conditionally in the same manner as .NET Framework (assuming the framework checked/inherited behavior from some system level setting), or to expose a configurable knob for people that need it to opt in.

Author:	mattpwhite
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[karelz]

Triage: Technically duplicate of #51701, but we may want to rethink this one ...

[mattpwhite]

Sorry, didn't notice that existing issue. That linked issue mentions that this is conforming to a spec. Which spec is that? https://datatracker.ietf.org/doc/html/rfc4559 makes no mention of including a port. That RFC is basically a retcon of IE behavior (came 6 years after IE shipped the feature), and IE doesn't do this by default. Chrome and Firefox don't do this, nor does curl.

[mattpwhite]

Thanks for addressing this, much appreciated.