Regression: `Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory` no longer allows the content root to be set programatically in some cases

[JeremySkinner]

Describe the bug

Hi, In .NET 5 it was possible to override the ConfigureWebHost method of WebApplicationFactory and call builder.UseContentRoot(".") when the content root can't be inferred (eg if the startup class isn't defined inside a web project).

In .NET 6, this is no longer possible as the SetContentRoot method now always calls GetContentRootFromFile. GetContentRootFromFile throws an exception rather than returning null if no content root is present in MvcTestingAppManifest.json.

This affects the integration tests of the FluentValidation project, which need to be able to set the content root programatically (the controller end points, startup class, and integration tests themselves are all in the same project).

I haven't looked into how the MvcTestingAppManifest.json file is generated, but it seems that it isn't populated with the test assembly itself, which is a problem if the Startup/entrypoint class is defined inside the test project, rather than a web project.

To Reproduce

• Create an integration test project
• Add the startup file to the integration test project (not to a separate web project)
• Create the test host using WebApplicationFactory to point to the Startup class that's inside the test project
• edit I've made a small sample project to reproduce: https://github.com/JeremySkinner/IntegrationTestingRegression

Here is the web application factory that I was using in .net 5:

public class WebAppFixture : WebApplicationFactory<Startup> {

		protected override void ConfigureWebHost(IWebHostBuilder builder) {
			builder.UseContentRoot(".");
		}
 //...
}

In .NET 5 this worked fine, but in .NET 6 ConfigureWebHost is never called as GetContentRootFromFile will always throw an exception before it gets to this point.

My hacky workaround is to delete the manfiest file in the ctor:

public WebAppFixture() {
	if (File.Exists("MvcTestingAppManifest.json")) {
		File.Delete("MvcTestingAppManifest.json");
	}
}

...and then GetContentRootFromFile will never execute, and the exception won't be thrown.

Recommendation

If GetContentRootFromFile could be modified to return null rather than throw an exception, then the previous behaviour would continue to work.

aspnetcore/src/Mvc/Mvc.Testing/src/WebApplicationFactory.cs

Lines 228 to 239 in 1395fa8

	private static string GetContentRootFromFile(string file)
	{
	var data = JsonSerializer.Deserialize<IDictionary<string, string>>(File.ReadAllBytes(file))!;
	var key = typeof(TEntryPoint).Assembly.GetName().FullName;
	if (!data.TryGetValue(key, out var contentRoot))
	{
	throw new KeyNotFoundException($"Could not find content root for project '{key}' in test manifest file '{file}'");
	}
	return (contentRoot == "~") ? AppContext.BaseDirectory : contentRoot;
	}

If the above method returned null, then the "old" (net5) codepath would continue to run, as it already handles the case when the content root is null:

aspnetcore/src/Mvc/Mvc.Testing/src/WebApplicationFactory.cs

Lines 208 to 226 in 1395fa8

	private void SetContentRoot(IWebHostBuilder builder)
	{
	if (SetContentRootFromSetting(builder))
	{
	return;
	}
	var fromFile = File.Exists("MvcTestingAppManifest.json");
	var contentRoot = fromFile ? GetContentRootFromFile("MvcTestingAppManifest.json") : GetContentRootFromAssembly();
	if (contentRoot != null)
	{
	builder.UseContentRoot(contentRoot);
	}
	else
	{
	builder.UseSolutionRelativeContentRoot(typeof(TEntryPoint).Assembly.GetName().Name!);
	}
	}

...and I believe adding an override in ConfigureWebHost would then continue to work.

Exceptions (if any)

Error Message:
   System.Collections.Generic.KeyNotFoundException : Could not find content root for project 'FluentValidation.Tests.AspNetCore, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' in test manifest file 'MvcTestingAppManifest.json'

Further technical details

• ASP.NET Core version: 6 preview 6
• Include the output of dotnet --info

.NET SDK (reflecting any global.json):
 Version:   6.0.100-preview.6.21355.2
 Commit:    7f8e0d76c0

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         ubuntu.20.04-x64
 Base Path:   /home/jskinner/.dotnet/sdk/6.0.100-preview.6.21355.2/

Host (useful for support):
  Version: 6.0.0-preview.6.21352.12
  Commit:  770d630b28

.NET SDKs installed:
  2.1.505 [/home/jskinner/.dotnet/sdk]
  3.1.408 [/home/jskinner/.dotnet/sdk]
  5.0.202 [/home/jskinner/.dotnet/sdk]
  6.0.100-preview.6.21355.2 [/home/jskinner/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.9 [/home/jskinner/.dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.9 [/home/jskinner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.14 [/home/jskinner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.5 [/home/jskinner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0-preview.6.21355.2 [/home/jskinner/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.9 [/home/jskinner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.14 [/home/jskinner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.5 [/home/jskinner/.dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.6.21352.12 [/home/jskinner/.dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

[JeremySkinner]

Here's a minimal sample project to reproduce: https://github.com/JeremySkinner/IntegrationTestingRegression

git clone https://github.com/JeremySkinner/IntegrationTestingRegression.git
cd IntegrationTestingRegression
dotnet test

Output:

Error Message:
   System.Collections.Generic.KeyNotFoundException : Could not find content root for project 'IntegrationTestingRegression, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' in test manifest file 'MvcTestingAppManifest.json'

[mkArtakMSFT]

@davidfowl have you been working on this area recently? Is this intentional?

[davidfowl]

Looks like it was caused by this change, @captainsafia might have more information.

[JeremySkinner]

Thanks @captainsafia!

[JeremySkinner]

Hi @captainsafia I'm still seeing this behaviour with preview 7 so I'm guessing this didn't make it in time for this release. Do you know which release the fix is targetted for? Thanks!

[martincostello]

Yeah the fix for this missed out on preview 7 (tag). It's in the RC1 daily builds.

https://github.com/dotnet/aspnetcore/commits/v6.0.0-preview.7.21378.6/src/Mvc/Mvc.Testing/src/WebApplicationFactory.cs