GHC Killed with out of memory when using generics

[jcristovao]

Aeson 0.10 seems to somehow have changed the way it handles generics, and thus crashes ghc.

Cabal file:

name:                a
version:             0.1.0.0
build-type:          Simple
cabal-version:       >=1.10

library
  exposed-modules:     Test
  build-depends:       base >=4.8 && <4.9, aeson >=0.10 && <0.11, iso3166-country-codes
  hs-source-dirs:      src
  default-language:    Haskell2010

iso3166-country-codes is a very simple module with a datatype which lists all countries. It derives Generic.
https://hackage.haskell.org/package/iso3166-country-codes-0.20140203.7/docs/Data-ISO3166_CountryCodes.html

The source code (src/Test.hs):

  module Test
    (
    CountryCode(..)
  )where

  import Data.Aeson
  import Data.ISO3166_CountryCodes

  instance ToJSON CountryCode

Crashes GHC (Killed) - memory consumption raises up to 4GB before crashing.
This did not happen with previous versions of Aeson.

PS: I'm using ghc 7.10.1

[cgadski]

I managed to build it successfully, but with a hideous peak of ~7 GB of memory usage.

[basvandijk]

This is very likely caused by GHC bug #5642.

The only thing I can't explain is why you didn't experience the problem on older versions of aeson.

[jgm]

I'm having the same issue.

Text.Pandoc.Definition (in pandoc-types) is a simple module that just defines some types and derives instances (including generic ToJSON and FromJSON instances).

When I try to build it on a 64-bit ubuntu VM with 2 GB RAM, using stack lts-4.0 (and ghc 7.10.3), ghc fails on this module with an "out of memory" error.

Reverting the aeson version by adding aeson: 0.8.0.2 to stack.yaml solves the problem.

This is a serious issue for me. Suddenly users are complaining that cabal install pandoc fails with a mysterious ExitFailure 1 error. I could put an upper-bound on aeson in pandoc-types.cabal, but then pandoc will be excluded from stackage. Is there any way to work around this or fix it in aeson? (The GHC bug does seem relevant, but I can confirm that the problem does not arise for earlier versions of aeson.)

[RyanGlScott]

Here's a couple of observations:

1. The memory usage for deriving Generic will likely be a bit better in GHC 8.0, since it uses a new -XDataKinds-based mechanism that doesn't generate any instances or empty datatypes (which I suspect are contributing to the high memory usage). The evidence I have for this is that GHC has a perf test that measures how many allocations a large deriving Generic statement takes. It substantially decreased after the aformentioned changes went in.
2. I'm not intimately familiar with the workings of Data.Aeson.Types.Generic, but one thing that strikes me is that all of the class functions related to Generic classes are marked INLINE. I've ran into memory issues myself when trying to do something similar, and it appears other people have too. One possible way to alleviate the memory burden is to remove the INLINE pragmas (for now, at least).

[RyanGlScott]

Pull request #335 addresses this.