EF9: SaveChanges() is significantly slower in .NET9 vs. .NET8 when using .ToJson() Mapping vs. PostgreSQL Legacy POCO mapping

[c5racing]

We use PostgreSQL and are in the process of migrating from .NET8 to .NET9 and in doing so, changing from the Legacy POCO mapping to the newer .ToJson() Mapping.

In the attached sample, there is no data actually being modified. In all cases, dbContext.ChangeTracker.HasChanges(); returns false.

.NET8, version 8.0.11
Legacy Poco Mapping: 0ms
.ToJson(): 300ms

.NET9, version 9.0.0
Legacy Poco Mapping: 0ms
.ToJson(): 2500ms

In both cases, .ToJson() takes longer; however, the difference between .NET8 and .NET9 is 10x more.

The sample references .NET9; however, can be changed to .NET8 for testing. It also relies on Docker for Testcontainers.PostgreSql

https://drive.google.com/file/d/1A_VoN9XgrogMEeevV7mtLh5c2s-Ds2rt/view?usp=drive_link

[roji]

The difference between 8.0 and 9.0 is likely part of the more general #35053 (not specifically related to JSON), though we should test this scenario specifically after the fixes to #35053 to make sure.

Then for the general difference between PG legacy POCO mapping and ToJson, some difference is definitely expected (since legacy POCO treats the whole document as a single property, no change tracking inside); but 300ms does seem quite excessive (any thoughts @ajcvickers?).

[ajcvickers]

Yep, seems slow. But our relational JSON processing code is very inefficient--multiple passes, use of DOM, etc.--so it's not too much of a surprise.

[maumar]

Problem is with List<Guid> property. ListOfValueTypesComparer (as well as ListOfNullableValueTypesComparer and ListOfReferenceTypesComparer) use ValueComparer.ConstructorExpression to build a new instance of valuecomparer every time it's called rather than use the instance passed as an argument (ValueComparer constant is not something we can represent in code for AOT case). This results in us creating 1000s of comparers, which then have their Equal(Guid Guid) delegate lazilly compiled (1000s of times) during save changes.

[maumar]

We should use EqualsExpression SnapshotExpression from the element comparer rather than comparer itself. So, instead of

private static bool Compare(IEnumerable<TElement>? a, IEnumerable<TElement>? b, ValueComparer<TElement> elementComparer)

we should use

private static bool Compare(IEnumerable<TElement>? a, IEnumerable<TElement>? b, Type elementComparerType, Func<TElement, TElement, bool> equalsFunc)