Read JSON from a character string

[jacobwilliams]

Would like to expand this library to also include the ability to read a JSON structure from a character string (currently, we can only read from a file).

[zbeekman]

That would be a nice feature. Writing to a character string would be nice too, but much more challenging to implement, I think.

[jacobwilliams]

Writing to a string is already supported (see json_print_to_string).

[zbeekman]

Yeah, it is, and I knew that too. A bit derpy today.

[jacobwilliams]

I'm working on this. When it's done, I'm curious to know if it would be more efficient if this was the default (i.e., the whole file is read into a character string, and then all further parsing is done using the string, rather than reading the file one character at a time). It will require some testing.

[zbeekman]

So I haven’t dug around the IO portion of the code in any serious way, but IO is a bit finicky to figure out how to optimize. First of all, different compiler vendors treat buffering differently. To enable buffering with ifort one needs to add -assume buffered_io when linking/compiling. In my experience, this almost always increases performance. There are also some environment variables which can be set at run time to control IO buffering with ifort, but I can’t remember what they are since they are only in the manual and not in the man page. You can control the size and number of IO buffers this way, and usually this helps speed things up.

Another way to improve IO performance is to use something like a double-buffered read with the asynchronous IO feature of F2003. This way you could amortize the IO time by doing work to parse a buffer already read into memory, while waiting for the next one to be filled. Steve Lionel told me that Intel’s implementation of asynchronous IO is thread safe, so it should be able to be used in MPI and OpenMP programs too. (I think it uses OS specific system calls for the IO stuff, not posix threads or OpenMP threads… not 100% sure.) This is often a good way to amortize the IO costs and hide the work being done while parsing. Depending on how the parser works, it might be hard to parse an incomplete portion of the file (possibly broken at an arbitrary new line) then pause and switch to the next buffer and continue. How this might interact with compiler supplied IO buffering is unclear to me, but I would guess that this method gives the best performance, and should not be combined with compiler enabled buffering.

[jacobwilliams]

Will close this issue once I test it further.

[jacobwilliams]

It seems to work fine.