How to use the original $PATH environment variable inside a stack execution

[jneira]

Hi! I recently discovered that stack is modifying the $PATH variable when you ran any command. More precisely it prepends to it {local-install-root}/bin and {snapshot-install-root}/bin.

It is interfering with an actual use case in haskell-ide-engine. We are using a script to perform the installation of the tool and one of the commands installs the appropiate version of cabal in local-bin-dir (that hopefully is in $PATH).

The issue is once you have installed it the first time, stack put it in {snapshot-install-root}/bin and all subsequent calls to the script using stack use that cached version of cabal and no the original executable in user $PATH (wherever it is).

That behaviour causes:

• If you delete the cabal executable in $PATH after installing it with the script, another invocation doesn't copy again the cabal executable, cause the version cached by stack is found (using Directory.findExecutable).
• All executions of the script that eventually execute cabal use that cached version and no the original one, so f.e. if you upgrade manually your cabal in $PATH to 3.0.0.0 the script will continue using the cached version, (currently 2.4.1.0)

The unique workaround i've got is to manually remove those paths from $PATH and run all cabal executions and Directory.findExecutable without them:

withoutStackCachedBinaries action = do
  mbPath <- liftIO (lookupEnv "PATH")

  case (mbPath, isRunFromStack) of

    (Just paths, True) -> do
      snapshotDir <- trimmedStdout <$> execStackShake ["path", "--snapshot-install-root"]
      localInstallDir <- trimmedStdout <$> execStackShake ["path", "--local-install-root"]

      let cacheBinPaths = [snapshotDir </> "bin", localInstallDir </> "bin"]
      let origPaths = removePathsContaining cacheBinPaths paths

      liftIO (setEnv "PATH" origPaths)
      a <- action
      liftIO (setEnv "PATH" paths)
      return a

    otherwise -> action

  where ....

Clearly it is an ugly and brittle hack so do you know another better way to accomplish it?
In case there is no one, could be added some way to optionally bypass the stack modifications of the $PATH variable?

(NOTE: i am aware that the modification of the $PATH variable is absolutely required to not break stack builds, optionally bypassing it only would make sense inside stack run script.hs or maybe stack exec whatever. A possible improvement could be save the original path to another environment variable with a specific name)

Stack version

$ stack --version
Version 2.1.3, Git revision 0fa51b9925decd937e4a993ad90cb686f88fa282 (7739 commits) x86_64 hpack-0.31.2

Method of installation

• Official binary, downloaded from stackage.org or fpcomplete's package repository

[qrilka]

@jneira maybe I'm missing here but it looks to me like your problem is caused by the following:
Stack can't clear cache for already installed executables. This could be resolved either by adding a flag to stack install which will not take already cached executable into account or some way to remove that executable from being cached by Stack. Does this sound correct to you or maybe I miss some case in the way you use Stack which could not be resolve by those 2 options?

[jneira]

@qrilka yeah, you have delimited it neatly, thanks. Only note that the problem is not in the build and install phase: we want that stack uses the cache for installing it (and no trigger a full build) but not at "runtime" (within stack run or stack script.hs and maybe within stack exec, although our actual use case would be covered by the first two) So the flag you mentioned could be added only to stack run and maybe stack exec.

[qrilka]

But that's sounds to be against how Stack is working: with stack install (which is actually stack build + copy of executable) you bring the cabal executable into implicit snapshot. As ghc-pkg doesn't manage executables, those are made available with Cabal's copy command with {snapshot-install-root}/bin as a destination. And after polluting user's implicit snapshot with cabal-install you want to undo this for stack script or stack exec. This sounds like a contradiction.
At the same time if we talk about unregistering cabal-install from a snapshot I think there is a problem that there seem to be no easy way to do that. With library packages there is a workaround to use ghc-pkg unregister but cabal-install contains only executable and deleting cabal from {snapshot-install-root}/bin makes Stack's view on snapshot contents inconsistent:

qrilka@qdesktop ~/ws/h/hatrace $ rm $(stack exec which cabal)
qrilka@qdesktop ~/ws/h/hatrace $ stack exec which cabal
/usr/bin/which: no cabal in (/home/qrilka/ws/h/hatrace/.stack-work/install/x86_64-linux-tinfo6/0fc8b77a62461097f7df5cae135a14f2fefc9b0db0c5959cd4bfe30a1eace3f2/8.6.3/bin:/home/qrilka/.stack/snapshots/x86_64-linux-tinfo6/0fc8b77a62461097f7df5cae135a14f2fefc9b0db0c5959cd4bfe30a1eace3f2/8.6.3/bin:/home/qrilka/.stack/compiler-tools/x86_64-linux-tinfo6/ghc-8.6.3/bin:/home/qrilka/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin:/home/qrilka/bin:/home/qrilka/.local/bin:/home/qrilka/.npm-packages/bin:/home/qrilka/.cask/bin:/home/qrilka/.nix-profile/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/opt/bin/:/usr/games/bin:/home/qrilka/.cargo/bin)
qrilka@qdesktop ~/ws/h/hatrace $ stack install cabal-install
Couldn't find executable cabal in directory /home/qrilka/.stack/snapshots/x86_64-linux-tinfo6/0fc8b77a62461097f7df5cae135a14f2fefc9b0db0c5959cd4bfe30a1eace3f2/8.6.3/bin/

[jneira]

This sounds like a contradiction.

Well, i agree that the most common use cases needs the actual behaviour. But i think the exposed one is reasonable too:

1. Run a shake script with stack that checks if a excutable is present in the (original) user $PATH.
   • We dont want that the script see the stack based binaries installed by previous script executions
2. Install it if it is not present (but honour the previous one), calling stack install inside the shake script
   • We want those calls to stack install sees the cached binaries as always
3. Make calls within the shake script to the executable present in the user original $PATH
   • and no the cached by stack cause the placed in the original $PATH could be a different, newer and preferred version

I think whatever script that want to manage external executables in a greceful way could hit this issue.

So i would add it only as a cli option (and no global config) cause it only makes sense for very concrete stack exections.

[qrilka]

I think this whole idea of a "script that want to manage external executables" is very suspicious. If you want an external executable then you should not use/pollute implicit snapshot you're using otherwise this executable is not external anymore but something managed by Stack so it makes perfect sense that it's use is governed by Stack itself. So logically imho you shouldn't use the same implicit snapshot when installing cabal-install if you want to have an external cabal executable. But that could get broken if somehow cabal-install will get installed into the implicit snapshot though I'm not sure if there is some reasonable way to get that.

[jneira]

So logically imho you shouldn't use the same implicit snapshot when installing cabal-install

Yeah, that could be a solution.

But that could get broken if somehow cabal-install will get installed into the implicit snapshot

is no there a way to make snapshots isolated to avoid that situation?

There is a --only-snapshot cli option, but no a --only-local one, that maybe could be useful in this case...

[qrilka]

Sorry for a very later reply.
Your question about isolated snapshots looks to be in conflict with your proposal about --only-local. Implicit snapshots are supposed to be isolated. The problematic aspect here is that an implicit snapshot includes only dependencies and changing a target doesn't change a snapshot.
--only-local will need some local copy of a package otherwise any Hackage package is treated the same way independent on whether it comes from Stackage snapshot or from extra-deps.

[jneira]

@qrilka No worries and thanks to take time to think about this somewhat edge case.

Your question about isolated snapshots looks to be in conflict with your proposal about --only-local. Implicit snapshots are supposed to be isolated

I see, thanks for point it out

--only-local will need some local copy of a package otherwise any Hackage package is treated the same way independent on whether it comes from Stackage snapshot or from extra-deps.

Then could it be doable? I am afraid that i dont know stack internals enough to give it a try in a reasonable amount of time. 😟

[jneira]

Hi, our use case does not exist anymore (we dont handle cabal inside a stack execution anymore) and it seems that the use case was pretty uncommon so i think we can close this.

@qrilka thanks for taking care of this and for your time

[qrilka]

@jneira just out of curiosity - what do you use now to install cabal for a Stack project?

[jneira]

we simply are separating cabal and stack executions of the script and make users install cabal themselves (he): haskell/haskell-ide-engine#1557

[qrilka]

Thanks