Configurable Path for `compiled`

[JaredCrean2]

Ref #30174 where it was noted that having source code and compiled code on the same filesystem may be problematic with the new atomic compilecache. Having an environment variable (separate from JULIA_DEPOT_PATH) to configure this would be good.

[vtjnash]

The JULIA_DEPOT_PATH already does this, so I'm not certain of the need for a duplicate mechanism. The first entry in the list gives the location for newly compiled code to be saved to. The list also gives an ordered search path for finding source files, and typically is approximately ordered from most-volatile first (perhaps temp files) to most-static (perhaps system files).

[JaredCrean2]

From the package manager docs for Depot Path:

The first entry is the “user depot” and should be writable by and owned by the current user. The user depot is where: registries are cloned, new package versions are installed, named environments are created and updated, package repos are cloned, newly compiled package image files are saved, log files are written, development packages are checked out by default, and global configuration data is saved.

It looks like any newly installed package will be installed to the first entry of the path, so making the first entry in JULIA_DEPOT_PATH point to a location on a non-persistent file system will results in source code getting install there too, which is not good.

[KristofferC]

Aren't the packages already installed? If not, could you just not push the depot path after they are installed but before they are loaded?

[JaredCrean2]

Having a static configuration is better than a dynamic one because it reduces the possibility of errors. For example, installing a few packages, running a simulation, installing one additional package and then having the one additional package magically disappear because it got installed to the non-persistent file system. It also jives better with the module system frequently used on clusters, where the person who creates the julia module can configure the depot and compiled paths for each user. This avoids the possibility of one user forgetting to set the compiled path and causing filesystem problems for everyone.