jstring_ being really slow #581
Description
I've run into the case where we try to parse JSON documents containing other serialized JSON documents, containing HTML and Base64 encoded strings. E.g. a JSON document containing HTML pages as well as base64 data with some additional info.
[{"html": "<html string>", "data": "<base64 string>", ... }]in a JSON envelope.
[{"date": "...", "user": "...", "payload": "<serialized json>", ... }]These documents vary from 1mb to 100mb. And anything above 25mb ends up being essentially
impossible with aeson to parse, as it consumes upwards of 10G memory and takes minutes to
process. All the ToJSON and FromJSON instances are generated with TH. And the profiling shows ~95% of the time to be spent in jstring_ enabling cffi seems to make no difference.