Is EF9 slower than EF8?

[wallyrion]

I observed that EF 9 is running noticeably slower and is allocating twice as much memory compared to its performance with EF 8. I'm trying to determine whether this performance degradation is an isolated exception or if there might be an issue with my benchmarking setup. Any insights would be appreciated

There are some benchmarks with BenchmarkDotnet

Method	Job	Runtime	UseSplitQuery	Mean	Error	StdDev	Ratio	RatioSD	Allocated	Alloc Ratio
GetCustomerById	.NET 8.0	.NET 8.0	False	1.688 ms	0.0368 ms	0.1038 ms	1.00	0.09	53.82 KB	1.00
GetCustomerById	.NET 9.0	.NET 9.0	False	2.516 ms	0.0474 ms	0.1059 ms	1.50	0.11	196.56 KB	3.65
GetCustomerById	.NET 8.0	.NET 8.0	True	2.867 ms	0.0598 ms	0.1705 ms	1.00	0.08	79.58 KB	1.00
GetCustomerById	.NET 9.0	.NET 9.0	True	3.114 ms	0.0732 ms	0.2136 ms	1.09	0.10	161.05 KB	2.02

Comparing the load testing metrics with k6. Performance degradation is also noticeable.

Metric	.NET 8	.NET 9
Total Requests	1,690,574	1,287,101
Requests per second	1,741.02	1,325.61
Data Received	13 GB	9.5 GB
Data Sent	220 MB	164 MB
HTTP Request Duration	avg=2.29s, p(90)=4.79s, p(95)=5.51s	avg=3.32s, p(90)=6.73s, p(95)=8.06s
Request Blocking	avg=6.02µs, p(90)=5.01µs, p(95)=5.81µs	avg=6.65µs, p(90)=5.15µs, p(95)=6.02µs
Request Receiving	avg=44.14µs, p(90)=61.38µs, p(95)=75.48µs	avg=42.62µs, p(90)=62.5µs, p(95)=76.01µs
Request Sending	avg=14.77µs, p(90)=16.12µs, p(95)=32.88µs	avg=13.8µs, p(90)=15.95µs, p(95)=29.79µs
Iteration Duration	avg=3.29s, p(90)=5.79s, p(95)=6.52s	avg=4.32s, p(90)=7.73s, p(95)=9.06s
Virtual Users (VUs)	max=10,000	max=10,000

Query for benchmarks.

 var result =
            await customerQuery.Select(customer => new CustomerDto
                {
                    Id = customer.Id,
                    Name = customer.Name,
                    Orders = customer.Orders.Select(order => new OrderDto
                    {
                        OrderId = order.Id,
                        OrderDate = order.OrderDate,
                        OrderItems = order.OrderItems.Select(oi => new OrderItemDto
                        {
                            OrderItemId = oi.Id,
                            ProductName = oi.ProductName,
                            Quantity = oi.Quantity,
                            Price = oi.Price
                        }).ToList()
                    }).ToList()
                })
                .FirstOrDefaultAsync(c => c.Id == customerId);

Benchmarking ToListAsync() (retrieving 10000 items into memory)

We can observe 7x performance degradation for both speed and memory allocation between EF8 and EF9

Method	Job	Runtime	Mean	Error	StdDev	Ratio	RatioSD	Allocated	Alloc Ratio
GetALlCustomers	.NET 8.0	.NET 8.0	1.788 s	0.0194 s	0.0172 s	1.00	0.01	272.78 MB	1.00
GetALlCustomers	.NET 9.0	.NET 9.0	7.926 s	0.0411 s	0.0384 s	4.43	0.05	1784.25 MB	6.54

Complete source code for the benchmarks

My repo

Include provider and version information

EF Core version:
Database provider Npgsql.EntityFrameworkCore.PostgreSQL (I tried to use SqlServer provider instead of Postgres and is shows the same results)
Version="8.0.10"
Version="9.0.0-rc.2.24474.1"
Target framework: (e.g. .NET 8.0 and .NET 9)
Operating system:
Windows 11 (for benchmarks) / Docker container (for load test)

[maumar]

The issue was introduced in Preview 4, specifically in this commit f0e88d4 (introduce liftable constant)

[matteomonizza]

In our project EF9 is more then 10x slower then previous

[maumar]

In the benchmark, considerable amount of time is now being spent in the ShaperProcessingExpressionVisitor.CompareIdentifiers (16.5% of total time, vs 0.7% in EF8). In EF8 we used to just pass a list of ValueComparers, but in EF9 we use List<Func<object, object, bool>>> which we construct manually:

        parentIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 
        outerIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 
        selfIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 

Before the method was just doing a simple call to ValueComparer.Equals, but now we invoke complex linq processing involving Linq.Expressions.Interpreter

[matteomonizza]

In EF9, a query in SQLite that perform in EF8 in one second, now takes 12 seconds.. we are investigating..

[maumar]

Benchmarks after removing invoke from the ValueComparer.ObjectEqualsExpression:

ef 8

Method	Async	Mean	Error	StdDev	Op/s	Gen0	Gen1	Allocated
PredicateMultipleIncludes	False	147.2 ms	2.63 ms	2.46 ms	6.793	4000.0000	3000.0000	26.24 MB
PredicateMultipleIncludes	True	159.1 ms	3.00 ms	2.95 ms	6.287	5500.0000	3000.0000	34.47 MB

ef 9 with Invoke

Method	Async	Mean	Error	StdDev	Op/s	Gen0	Gen1	Allocated
PredicateMultipleIncludes	False	322.6 ms	0.97 ms	0.86 ms	3.099	13000.0000	6000.0000	79.48 MB
PredicateMultipleIncludes	True	344.9 ms	6.79 ms	6.67 ms	2.899	14000.0000	7000.0000	87.72 MB

ef 9 without Invoke

Method	Async	Mean	Error	StdDev	Op/s	Gen0	Gen1	Allocated
PredicateMultipleIncludes	False	242.8 ms	2.39 ms	2.12 ms	4.119	8000.0000	5000.0000	51.69 MB
PredicateMultipleIncludes	True	263.4 ms	2.21 ms	2.06 ms	3.797	10000.0000	9000.0000	59.93 MB

Significant improvement, but still doesn't fully address the problem. Looking for more...