SNAT Port Exhaustion, Microsoft.AspNetCore.Identity.UserManager Async Calls Are blocking

[sbkaspersen]

Describe the bug

Some methods advertised as being async are in fact not.

Our application is using MSI authentication, and uses a db connection interceptor to set the access token on connection open. when we first enabled MSI authentication or application fell over at peak due to SNAT port exhaustion.

We eventually found the cause was due to workers being blocked on opening a db connection in a synchronous call even though the expectation was it would be async. The main culprit for our case was UserManager.FindByEmailAsync

Methods used by our application that exhibit this behaviour include

1. CreateAsync
2. FindByEmailAsync
3. UpdateAsync
4. ResetPasswordAsync
5. RemoveClaimsAsync
6. AddToRoleAsync

If you set RequireUniqueEmail to off then some if not all of the above do not use the blocking call.

To Reproduce

https://github.com/sbkaspersen/jubilant-sniffle

Further technical details

• ASP.NET Core version 3.1.3
• Include the output of dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.1.302
 Commit:    41faccf259

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.302\

Host (useful for support):
  Version: 3.1.6
  Commit:  3acd9b0cd1

.NET Core SDKs installed:
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• VS 2019 Enterprise

[davidfowl]

This looks like an entity framework issue as that's where you're currently plugging in the MSI auth logic.

[davidfowl]

@ajcvickers wheres the appropriate way to plug in MSI?

cc @blowdart as you were looking at MSI

[blowdart]

@sbkaspersen Can you share which version of the MSI libraries you are using, and how you're wiring them up?

[sbkaspersen]

@blowdart
We are running the webapp in azure. Using Microsoft.Azure.Services.AppAuthentication 1.4.0. AzureServiceTokenProvider is injected into the DB Conneciton Interceptor (commented out code in the repro).

We have worked around the issue by bypassing calls to the mentioned methods (Instead using the db context directly)

While MSI is symptomatic of the bug, the actual bug I believe is the Task.FromResult here. At least for FindByEmail., I suspect the other ones we found are related as they are only reproducible if
RequireUniqueEmail is true.

[smitpatel]

Regardless of it being root cause, it is an error in identity. 9ebabc1
This commit changed it to sync version when EF Core query pipeline was being rewritten.
But when removing work-around it did not revert back to async version 839cf89

[davidfowl]

I see so this line of code https://github.com/sbkaspersen/jubilant-sniffle/blob/4e31cf5b9c899da525f37bc600cbb25f0a0821ad/UserManagerAsyncTest.cs#L19 is irrelevant?

[sbkaspersen]

@davidfowl, That was the line of code causing all our worker threads to block.

Since we were using async calls our expectation was that https://github.com/sbkaspersen/jubilant-sniffle/blob/4e31cf5b9c899da525f37bc600cbb25f0a0821ad/UserManagerAsyncTest.cs#L32 would be entered and not L19 as is the current case.

When we use our own async dbcontext calls instead of UserManager we are only entering ConnectionOpeningAsync which does not block the worker threads.

[davidfowl]

Got it, so this does look like an identity issue.

[HaoK]

Oops yeah, missing an Async, wonder how far back this goes

[HaoK]

All the way back to 2.1 it looks like, how far back should we patch @ajcvickers @blowdart ?