Document why hls binaries are statically linked (and track ghc changes to allow loading dynamic libs within a static executable)

[nh2]

I learned from @mpickering that HLS distributes GHC binaries with DYNAMIC_GHC_PROGRAMS=YES, and that this creates various problems for TemplateHaskell regarding dynamic linking.

I am interested in this topic because of NixOS/nixpkgs#129245.

I was trying to find documentation on the fact that, or why, HLS does that, but couldn't find any.

Could that be written down somewhere? It would make users' and packagers' life easier.

---

On the question itself: Would DYNAMIC_GHC_PROGRAMS=YES be more problematic to distribute somwhow? It appears to me that if e.g. the VSCode plugin downloader can download GHC, it could also download some .so files with it.

[pepeiborra]

There seems to be some confusion here. DYNAMIC_GHC_PROGRAMS is a ghc flag and has nothing to do with HLS.

I think you are referring to the fact that distribution HLS binaries are linked statically, that is, both Haskell packages and external libraries (modulo glibc) are statically linked in a single binary. This is done for ease of distribution.

The problem comes because the ghc package is linked statically, which means the runtime the RTS linker will be used to dynamically load packages and their external dependencies, as documented here. Unfortunately the RTS linker is far from perfect and has issues in all platforms.

GHC binaries switched to DYNAMIC_GHC_PROGRAMS=YES in 7.8/7.10, except on Windows where I think they are still statically linked. This was done for good reason: TH and GHCi are able to use dlopen to load packages and libraries.

Ideally, the choice of linker/loader would be decoupled from whether the GHC package is linked statically or dynamically. I remember asking @bgamari at some point to write down what needs to be done in a ticket for future reference.

[nh2]

There seems to be some confusion here. DYNAMIC_GHC_PROGRAMS is a ghc flag and has nothing to do with HLS.

I think you are referring to the fact that distribution HLS binaries are linked statically

Yes, that's what I meant with "HLS distributes GHC binaries with DYNAMIC_GHC_PROGRAMS=YES".

as documented here

Thanks for linking me that page. It occurred to me now that that the "What happens if we build a static executable and link in the GHC package?" mentioned in there also applies to the ghc binary itself.

This is done for ease of distribution.

Could you elaborate in which way it makes distribution easier? Is it just have less files around, or are there othere benefits?

Thanks!

[pepeiborra]

There seems to be some confusion here. DYNAMIC_GHC_PROGRAMS is a ghc flag and has nothing to do with HLS.
I think you are referring to the fact that distribution HLS binaries are linked statically

Yes, that's what I meant with "HLS distributes GHC binaries with DYNAMIC_GHC_PROGRAMS=YES".

as documented here

Thanks for linking me that page. It occurred to me now that that the "What happens if we build a static executable and link in the GHC package?" mentioned in there also applies to the ghc binary itself.

This is done for ease of distribution.

Could you elaborate in which way it makes distribution easier? Is it just have less files around, or are there othere benefits?

Thanks!

Yes

[nh2]

Ah sorry, I just noticed what introduced confusion. I wrote:

I learned from @mpickering that HLS distributes GHC binaries with DYNAMIC_GHC_PROGRAMS=YES,

but I meant to write:

I learned from @mpickering that HLS distributes GHC binaries with DYNAMIC_GHC_PROGRAMS=NO

(that the GHC binaries distributed by HLS use NO).

If the main benefit to HLS of using DYNAMIC_GHC_PROGRAMS=NO is "less files", wouldn't the fact that TH does not work well with it be a big motivator to switch to YES instead, like the official GHC bindists?

More files seem like an OK tradeoff to make vs wrestling dynamic loader problems.

[pepeiborra]

I simply don't have the expertise to set up dynamically linked binaries for distribution (other than a simple Docker image). It would be awesome if someone else can make that happen.

[hasufell]

I consider this a GHC bug and I'd advise to pursue an upstream fix before going down the road of dynamic linking on linux. Static linking is a huge win for compatibility issues. Ultimately, we want GHC statically linked, so stack, ghcup etc don't have to guess which bindist is compatible with distro X version Y.

[adamse]

@hasufell which part do you consider a ghc bug?

[hasufell]

@adamse

As @pepeiborra said

Ideally, the choice of linker/loader would be decoupled from whether the GHC package is linked statically or dynamically. I remember asking @bgamari at some point to write down what needs to be done in a ticket for future reference.

[adamse]

Ah, that's a good point. I've thought about a little bit, and I think it should be possible :)

[pepeiborra]

Are you planning to work on this Adam? That would be awesome