Memory Leaks starting with DotNet 6 Preview 5 in Blazor Server Statically-Rendered Components

[fitdev]

Describe the bug

After upgrading from DotNet 6 Preview 4 to Preview 5, and then, now to Preview 6, I noticed my website memory usage going up with time, whereas on Preview 4 it pretty much stayed the same for days, now it was adding like 100MB every half-hour, even though no changes have been made (other than recompile for Preview 6) and the website load (users) remained the same.

I suspected a memory leak, and so collected GC Heap dumps every few hours. What I saw was this:

Then, after running for 1 more hour:

Clearly there was some change introduced with Preview 5, which is also in Preview 6, as both of those exhibit the same behavior. Preview 4, however works fine.

What is interesting is that all of my Razor components are exposed via a single cshtml view with basically one line of code:

<component type="Model.RazorComponentType" render-mode="Static" param-Model="Model" />

That is all Razor components are rendered statically on the Server.

What I tried:

• x86 vs x64 made no difference.
• UseRazorSourceGenerator in MSBuild Project file to either generate mywebsite.Views.dll or not made no difference.

My Code does use quite elaborate component inheritance, as well as manually constructed components (entirely in code using RenderTreeBuilder). But the point is I have not done any changes to code, and what previously worked in DotNet 5 and DotNet 6 Prview 4, no longer does and lead to this memory leak.

Further technical details

• ASP.NET Core version: 6.0.0-preview.6.21352.12
• Include the output of dotnet --info:

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100-preview.6.21355.2\

Host (useful for support):
  Version: 6.0.0-preview.6.21352.12
  Commit:  770d630b28

.NET SDKs installed:
  2.1.520 [C:\Program Files\dotnet\sdk]
  5.0.400-preview.21328.4 [C:\Program Files\dotnet\sdk]
  6.0.100-preview.6.21355.2 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0-preview.6.21355.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.6.21352.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-preview.6.21353.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version: VS 2022 Preview 2 (though project was built and deployed using DotNet CLI)

[javiercn]

@fitdev thanks for contacting us.

Have you tried forcing a Gen 2 GC and checking if the objects are rooted after that?

Are you able to make the memory grow without it being eventually freed by the GC?

[fitdev]

@javiercn Thank you for looking into this.

Prior to both of the snapshots above there have been on the order of 100 Gen0 and Gen1 collections. And maybe around 10-20 Gen2 collections. So, to answer your 2nd question, Yes the memory definitely grows and is not being freed by the GC.

I should also mention that I tried Server vs Workstation GC flavors with mo difference.

I am not sure what good forcing a GC will do, considering during the lifetime there are many GCs happening that have no effect on the matter (i.e. memory usage grows).

Also, perhaps I should mention that under Paths to Root cycles have been detected:

[javiercn]

@fitdev triggering a Gen 2 GC collection helps with establishing whether something is holding on to the objects, most GC collections don't clear all the objects.

Can you check what things are rooting those objects? The fact that you see a bunch of Blazor types on the memory profile doesn't necessarily imply that its an issue in the framework, it can be something on your app causing those objects to stay in memory.

We would need a bit more info to make progress here since this can be something caused by your app and we don't have a good way to know:

• If you can create a minimal repro project that has this behavior that would work.
• Alternatively, check the GC roots for your app and trace the path starting from the framework code that is holding on to that memory.

[fitdev]

I will try forcing a Gen2 collection and see what I get, as you suggest.

But what could have changed between Preview 4 and Preview 5 that caused this behavior to start happening?

[javiercn]

Also, its fundamental that you run the app in Release and the publish output, not from Visual Studio

[fitdev]

I don't use VS to publish. I use CLI by running: dotnet build src\Website.csproj /p:DeployOnBuild=true /p:PublishProfile=MyProfile

With these options:

    <WebPublishMethod>MSDeploy</WebPublishMethod>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SelfContained>true</SelfContained>
    <MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
    <EnableMSDeployBackup>True</EnableMSDeployBackup>
    <TargetFramework>net6.0</TargetFramework>
    <RuntimeIdentifier>win-x86</RuntimeIdentifier>
    <EnvironmentName>Production</EnvironmentName>

[javiercn]

@fitdev that's fine, I just want to make sure that you are not running a debug build nor from in development mode. (dotnet run and similar).

Just dotnet your.dll against the Release build of what you generate to make sure there are no dev features that can potentially interfere like hot reload.

[fitdev]

What do you suggest I use to:

Alternatively, check the GC roots for your app and trace the path starting from the framework code that is holding on to that memory.

I cannot seem to find this functionality in the Managed Heap view.

[javiercn]

You can take a memory dump and use you can use dotnet dump or windbg to do that as described here

[fitdev]

Here's the output of gcroot -all ran on a random instance of Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder within the context of dotnet-dump tool.

Statistics:
              MT    Count    TotalSize Class Name
00007ffa1c3c2348     3572       200032 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
Total 3572 objects
> gcroot -all 00000168123cc250
HandleTable:
    000001680D201358 (strong handle)
    -> 0000016828319C10 System.Object[]
    -> 000001681223DF30 System.Action
    -> 0000016811CBBEF8 System.Object[]
    -> 000001681223DEF0 System.Action
    -> 000001681223D878 Microsoft.AspNetCore.Components.Rendering.HtmlRenderer
    -> 000001681223E7F0 System.Collections.Generic.List`1[[System.ValueTuple`2[[Microsoft.AspNetCore.Components.Rendering.ComponentState, Microsoft.AspNetCore.Components],[Microsoft.AspNetCore.Components.ParameterView, Microsoft.AspNetCor
Components]], System.Private.CoreLib]]
    -> 000001681223E8C8 System.ValueTuple`2[[Microsoft.AspNetCore.Components.Rendering.ComponentState, Microsoft.AspNetCore.Components],[Microsoft.AspNetCore.Components.ParameterView, Microsoft.AspNetCore.Components]][]
    -> 000001681223E5C8 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 000001681223E730 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 000001681223E768 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812240960 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168122410D8 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122411B0 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122411E8 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812242708 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812242CC0 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812242D98 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812242DD0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812242E48 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812243400 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122434D8 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812243510 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812243588 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812243D80 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812243EC0 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812243EF8 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812243F70 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812244510 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122445E8 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812244620 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122446B0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812244C50 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812244D28 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812244D60 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812244DF0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812245548 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812245688 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122456C0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122459D0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812245F70 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812246048 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812246080 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812246110 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168122466B0 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812246788 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122467C0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812246E60 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812247C60 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812247D38 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812247D70 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812253368 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812257040 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812257118 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812257150 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122737C8 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812275010 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122750E8 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812275120 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812241340 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812279020 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812279160 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812279198 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122758B0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168122A18E0 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122A1B00 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122A1B38 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122BEA18 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168122BF070 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122BF1B0 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122BF1E8 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168122DDB78 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168122DECD0 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168122DEE10 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168122DEE48 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812315D20 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 0000016812317F48 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 0000016812318088 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 00000168123180C0 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 0000016812363D70 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168123644A0 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168123645E0 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder
    -> 0000016812364618 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrameArrayBuilder
    -> 00000168123CB780 Microsoft.AspNetCore.Components.RenderTree.RenderTreeFrame[]
    -> 00000168123CC110 Microsoft.AspNetCore.Components.Rendering.ComponentState
    -> 00000168123CC250 Microsoft.AspNetCore.Components.Rendering.RenderTreeBuilder

Found 1 roots.

Not sure what to make of it. Other instances look pretty similar.

[fitdev]

I also noticed that the number of rooted Microsoft.AspNetCore.Components.Rendering.HtmlRenderers keeps going up, despite the fact that they should have been disposed of and collected. In my case the number of Microsoft.AspNetCore.Components.Rendering.HtmlRenderers that are kept alive very closely corresponds to the number of page views there have been during app's lifetime. So it is as if, every page view leaves behind an uncollected renderer.

For instance:

> dumpheap -mt 00007ffa1b2535c0
         Address               MT     Size
0000016810c03528 00007ffa1b2535c0      144
0000016810e86ec8 00007ffa1b2535c0      144
00000168112441b0 00007ffa1b2535c0      144
0000016811cbb6e8 00007ffa1b2535c0      144
0000016811d6e6c8 00007ffa1b2535c0      144
00000168121c4660 00007ffa1b2535c0      144
000001681223d878 00007ffa1b2535c0      144

Statistics:
              MT    Count    TotalSize Class Name
00007ffa1b2535c0        7         1008 Microsoft.AspNetCore.Components.Rendering.HtmlRenderer
Total 7 objects
> gcroot -all 0000016811d6e6c8
HandleTable:
    000001680D201358 (strong handle)
    -> 0000016828319C10 System.Object[]
    -> 000001681223DF30 System.Action
    -> 0000016811CBBEF8 System.Object[]
    -> 0000016811D6ED40 System.Action
    -> 0000016811D6E6C8 Microsoft.AspNetCore.Components.Rendering.HtmlRenderer

Found 1 roots.

[javiercn]

I too a quick look just for sanity and I don't see anything wrong. These objects get created via DI and are registered in the container, which manages their lifecycle.

aspnetcore/src/Mvc/Mvc.ViewFeatures/src/DependencyInjection/MvcViewFeaturesMvcCoreBuilderExtensions.cs

Lines 236 to 238 in 52eff90

	services.TryAddScoped<IComponentRenderer, ComponentRenderer>();
	services.TryAddScoped<StaticComponentRenderer>();
	services.TryAddScoped<HtmlRenderer>();

When the scope for the request goes away the objects should go away too, unless there is something holding on to them. From what I can tell, we don't hold on to them anywhere outside of that chain, so it likely points to something in your app. Either that or DI is completely broken, which I doubt.

Essentially you need to check two things:

• Are those objects being collected (by a forced gen 2 GC triggered with GC.Collect(2, force: true))
  • If they are, its not a memory leak
• If they are not, can you send requests up until you get an Out of memory exception?
  • If not, its not a memory leak
  • If you do then, we need to find out what is holding on to those objects:
    • Does the behavior repro on an app with just framework code?
    • Memory leak
    • Potential memory leak, but unlikely to be on the framework.

In the end, we need a chain of objects that starts from a static field or object on the stack that keeps accumulating references to these objects to conclude that there is a memory leak.

[fitdev]

Thank you for your suggestions. I will investigate more. But as for

Are those objects being collected (by a forced gen 2 GC triggered with GC.Collect(2, force: true))

my understanding is that they are not collected, since I used dotnet-gcdump to get heaps, and it forces full collection in order to produce the heap dump.