Add ability to run tests from cached initial conditions

[mark-petersen]

Right now, any tests run in compass must first generate their own mesh and initial condition files. This is fine for QU240, but for other global resolutions the init step can be lengthy, so it is more practical to point to cached initial condition files on disk. We could have a cached location on each supported machine (most already do), and compass could download files if they are not already there. We could point to E3SM files already here:
https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/
or add files here:
https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/mpas-ocean/

My motivation here is performance testing at varying resolutions, but this could be useful for any test or stand-alone simulation at resolutions higher that QU240.

[xylar]

@mark-petersen, looked into this a bit. I'll write up my thoughts quickly before I head to bed but will think more about this over the weekend.

It seems like there are 2 tracks.

The first seems to be what you had in mind. We could support initial conditions and meshes we already have in https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/. None of these were generated with the compass package but it should be able to perform forward runs with them. in compass with some tweaking. There are various issues with this. The current global_ocean approach assumes that the mesh is one of the ones defined in the compass.ocean.tests.global_ocean.mesh package. This is useful because there is a logical place to put metadata about the mesh as well as namelists and streams files needed for forward runs with various time integrators. (This is also where you define the mesh resolution, but that's not needed for cached initial conditions.) It will be hard to fit legacy mesh names into this approach in a clean way.

The second approach would be create all the cached initial conditions from scratch in compass and upload them to https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/mpas-ocean/. We would only support meshes that can also be created in compass so there would be a package for each in compass.ocean.tests.global_ocean.mesh, including the streams, namelists, etc. needed for forward runs. We would make these cached files adhere to a strict naming convention so it is clear what you are getting (and there would be a date stamp, allowing for updated versions as needed). This approach would require adding several more mesh resolutions to compass for your purposes.

I have a pretty strong preference for the second approach. Even so, I think there will be cases where we want to be able to use the first approach. For the purposes of generating mapping files and region masks, I have created a single test case that I can point to an existing mesh file and give a mesh name. This has been highly useful for producing mapping files for existing meshes where we either did not produce the masks when the mesh was generated or we needed to update them because of new region definitions. That test case is described here:
https://mpas-dev.github.io/compass/latest/users_guide/ocean/test_groups/global_ocean.html#make-diagnostic-files-test-case
(I seem to have missed adding it to the Developer's Guide for some reason.) I had initially thought we might be able to do something similar for cached initial conditions but it seems like the complexity of keeping track of namelist and steams files as well as other details about these files would require something quite a bit more complex than MakeDiagnosticFiles.