Discussion: unifying to fewer GitHub repos

[natemcmaster]

This is a discussion item for aspnet/Announcements#320. Please see the announcement for details.

Status

Repo	Code moved	Issues moved	Repo archived
AADIntegration	✓	✓	✓
Antiforgery	✓	✓	✓
AuthSamples	✓	✓	✓
AzureIntegration	✓	✓	✓
BasicMiddleware	✓	✓	✓
Caching	✓	✓	✓
Common	✓	✓	✓
Configuration	✓	✓	✓
CORS	✓	✓	✓
DataProtection	✓	✓	✓
DependencyInjection	✓	✓	✓
Diagnostics	✓	✓	✓
DotNetTools	✓	✓	✓
EventNotification	✓	✓	✓
FileSystem	✓	✓	✓
HtmlAbstractions	✓	✓	✓
HttpAbstractions	✓	✓	✓
HttpClientFactory	✓	✓	✓
HttpSysServer	✓	✓	✓
Hosting	✓	✓	✓
Identity	✓	✓	✓
IISIntegration	✓	✓	✓
JavaScriptServices	✓	✓	✓
JsonPatch	✓	✓	✓
KestrelHttpServer	✓	✓	✓
Localization	✓	✓	✓
Logging	✓	✓	✓
MetaPackages	✓	✓	✓
Microsoft.Data.Sqlite	✓	✓	✓
MusicStore	✓	✓	✓
Mvc	✓	✓	✓
MvcPrecompilation	✓	✓	✓
Options	✓	✓	✓
Proxy	✓	✓	✓
Razor	✓	✓	✓
ResponseCaching	✓	✓	✓
Routing	✓	✓	✓
Security	✓	✓	✓
ServerTests	✓	✓	✓
Session	✓	✓	✓
SignalR	✓	✓	✓
StaticFiles	✓	✓	✓
Templating	✓	✓	✓
Testing	✓	✓	✓
Universe	✓	✓	✓
WebHooks	✓	✓	✓
WebSockets	✓	✓	✓

[poke]

What will this mean for the solution files? Will there only be a single solution with all projects in that repo?

I assume this will make it a lot easier to contribute since we will no longer have to cross reference projects manually to make things compile but at the same time it will also make things a lot more confusing considering the high number of components in the project.

I do have some concerns about the Extensions projects though since these always felt very separate and individual to me, so merging unrelated things into a single repository does feel a bit weird. Especially considering that usually Git practices often encourage smaller repositories over large ones.

That being said, I applaud your approach to combining the repositories by merging the existing histories. I do consider keeping the original Git objects intact very important, so this is a very good move!

[Tragetaschen]

Since the existing repos all follow a very, very similar structure, are you going to extract/replay the git history for subfolders like src into the merged projects?

[Tragetaschen]

Is there a way to "Subscribe" to parts of the merged projects? For example, I'm subscribed to SignalR and Kestrel, but not to MVC or Razor

[Tratcher]

@poke We'll start by merging repos as subfolders with their existing structure inside, no changes to sln or other project files. Long term we don't plan on merging slns, VS can barely open them as is, but we will plan to unify some of the other build infrastructure.

[natemcmaster]

What will this mean for the solution files? Will there only be a single solution with all projects in that repo?

As Chris said, there won't be one big .sln file for all projects in larger repos. Despite our efforts, VS can't handle 700+ .csproj files yet. So, instead, we'll have multiple solutions. Each represents a "family" or workload of projects within the repo, such as Kestrel.sln, Mvc.sln, ChrisRossDoesHttp2Work.sln. Some of our larger repos, such as EF Core, already do this.

I do have some concerns about the Extensions projects though since these always felt very separate and individual to me

For what it's worth, this isn't changing the nature of the product. I view this as an update to our code-splitting strategy which reflects the reality of how the product ships. We ship all of those extensions packages on the same cadence using the same rules for patching, servicing policies, versioning, etc. Consumers should not be affected by reorganizing source code.

Especially considering that usually Git practices often encourage smaller repositories over large ones.

That's what started us down the road to where we are now: 53 git repos, some of which only produce 1 assembly. As the number of repos has grown over time, it has become increasingly expensive to manage them all. IMO the ideal number of git repos is somewhere ≥ 1 and ≤ # of devs working on a team.

Since the repos all follow a very, very similar structure, are you going to extract/replay the git history for subfolders like src into the merged projects?

Not exactly. We're going to be doing some reorganization of folders and then merging unrelated histories into the same tree. I'm playing with fun git things right now like octopus and subtree merges to find one that makes git log and git blame usable, if possible.

Is there a way to "Subscribe" to parts of the merged projects?

This is a GitHub notification limitation that regrettably has no good workarounds yet as far as I know. We'll use labels on issue trackers to group issues based on product area, like SignalR and Kestrel. Personally, I've stopped using email notifications and use octobox instead. I made a PR to this project a while ago to support filtering on labels.

[poke]

We'll start by merging repos as subfolders with their existing structure inside

Wait, does this mean that the repository will basically look like this then?

/
  Kestrel/
    src/
    test/
    Kestrel.sln
  Mvc/
    src/
    test/
    Mvc.sln
  …

Or will all the existing directories basically be merged, so that e.g. a current /src/ Microsoft.AspNetCore.Mvc.Core/ will still be at exact that same path in the new repo?

This has a huge impact on compatibility of the existing Git objects and basically decides on whether you will have to rewrite all changes (creating new objects for pretty much everything, breaking compatibility), or whether the existing changes are just being merged (retaining the full history without any changes).

I would very much like if e.g. a commit 956441aa68ce7516bce69fda616a0fe060c83882 that currently exists in Mvc, would still exist as an identical commit 956441aa68ce7516bce69fda616a0fe060c83882 in the new Home repo.

---

We ship all of those extensions packages on the same cadence using the same rules for patching, servicing policies, versioning, etc. Consumers should not be affected by reorganizing source code.

It does have an effect on everything related to issues though. Compared to the Home repo (which does have united issues for multiple repos for quite a while now), the extension repos feel very clean and focused.

But I totally do understand your motivations, especially considering how often you had cross-repo changes in the past that needed to be managed and timed properly.

[Tratcher]

Yes, it will start with folders like Kestrel/src, etc.. We are looking at how to preserve the commit history despite the observed git move. Apparently the information is all there and the main challenge is that individual tools like git blame do not traverse it fully.

[poke]

Hmm 😕 I’m not too happy with that then.

But you could achieve this nicely by adding each individual repo as a remote and merging that in directly (so that the contents are at the root), and then sorting them into subfolders in an additional commit. That way you should be able to keep the full individual histories untouched while also having log/blame continue to work because you are only doing a file move afterwards.

[Tratcher]

There's quite a few top level files that would conflict if we did that. The reverse might work though, changing the structure in the old repo and then merge it.

[TheSamsterZA]

What about, say, the Docs repositories (including the ones specifically for things like Entity Framework Core and Visual Studio Web)?

I assume a repo like Blazor would be moved under aspnet/AspLabs?

[Eilon]

@TheSamsterZA this plan is focused around the products that ship as part of .NET Core, so this will have no effect on Docs, Blazor, or other projects.

[aharpervc]

This is a GitHub notification limitation that regrettably has no good workarounds yet as far as I know. We'll use labels on issue trackers to group issues based on product area, like SignalR and Kestrel.

As someone who enjoys observing progress from the outside (not a contributor), this super sucks. I love watching the performance optimization PRs and discussion going on with Kestrel, and now that's not going to be possible due to being drowned out with a bunch of noise and other things I'm not interested in.