Submission client legibility and maintainability refactors

[mccheah]

Some of the recent work on the submission client has revealed that it can be difficult to add new features to it due to the way it is currently designed. This issue is for an open-ended discussion about how we could architect this better.

For background, the first version of the submission client was a monolithic design with a single class that handled almost all of the logic. The class was not unit tested and over time it became increasingly more difficult to refactor the code to make it unit-testable.

In submission v2 we attempted to fix this problem by breaking down the submission logic into multiple modules, where each module would be responsible for at most 1-2 functions.

Some of the breaking out had some consequences though:

(1) It's not clear how the different pieces link together and if they are related in some way. We tried to solve this by using the DriverInitContainerComponentsProvider as a centralized place to create all of the components.
(2) There are some sets of components, primarily around sending resources to the resource staging server, that only make sense to have all of them defined or none of them at all. There's a few places where the for...yield Scala feature is being used to encode this, but in reality they should all be packaged in higher-level structures where we can use an Option of the entire bundle and define all of the pieces or none of them cohesively.
(3) TLS and authentication management is fragmented across multiple classes, as illustrated in #347.

We still want to keep the modularity and ease of testing that we have right now, but it would be good if we could get some ideas for how to address these three issues. Unlike the transition from V1 to V2 we aim to keep the end-user interactions entirely untouched.

[mccheah]

@ifilonenko @chenchun for thoughts as you both have worked on this code recently. Are there any other things to consider aside from the three mentioned above?

@foxish @ash211 @erikerlandson @kimoonkim also.

[erikerlandson]

I think the "case class with copy method" builder pattern is generally effective, and case classes are easy to understand as objects representing configurations.

Some of the way this logic plays out does remind me of patterns using free monads and for-comprehensions, however working with those patterns isn't necessarily intuitive, and I'd worry about readability for the community. They can be useful for building up sequences of actions, where some sub-sequences may or may not be executed depending on the user's input. (an example would be python-related submission logic, whose steps only apply for python, or security logic that executes or not depending on what security config was applied)