Architecture document

[erikerlandson]

I don't think one this is a blocker for the 2.2 release, but upstream might be interested once we announce.

[mccheah]

I'm writing up some documents on the submission client now but might appreciate some input on the other aspects of the design like the scheduler and the external shuffle service.

[erikerlandson]

It seems like a lot of scheduler and shuffle service architecture is already covered in the Design Proposal, I wonder if we could just refer to that, and build out the Design Proposal if any new content on those is warranted?

[mccheah]

We'd probably want some more in-depth breakdown of the architecture, but KubernetesClusterSchedulerBackend is due for a refactor to split apart more components anyways.

[erikerlandson]

refactoring that seems like an "after 2.2" kind of project

[mccheah]

+1 - I guess our first pass at the architecture discourse for the scheduler backend might be higher level to account for the fact that things might change. At any rate we'll be discussing more from a fundamental design standpoint - e.g. "Here's how we rate limit executor requests" - rather than specific classes I suppose.