Runtime speed improvements

[jacobwilliams]

Need to do some speed tests, and also see if there are any changes that can be made to speed up the code. The file reading and/or parsing seems rather slow. See also #37.

[zbeekman]

Are you using big.json to do the testing?

[jacobwilliams]

Yes, see test case 9. I've also got some larger ones that I haven't committed.

[zbeekman]

ok, I was thinking a good place to start would be to run some of the tests through gprof and gprof2dot. (This will create pretty call graphs with timing info.) gprof doesn’t work on mac, so I may muck around and see if I can get travis-ci to do it for me. The timings won’t be great since it’s running in a busy VM, but it’s at least a starting point. If you want to muck around with it, you can install gprof2dot with sudo pip install gprof2dot (you’ll need graphviz too) and then look at the gprof2dot homepage for more info: https://code.google.com/p/jrfonseca/wiki/Gprof2Dot#gprof

[zbeekman]

Hopefully with some profiling, we can get a better understanding if it is #37, #38 or IO that is slowing things down.

[szaghi]

@jacobwilliams and @zbeekman
In my modest opinion, you that are not so poor Fortran man can also exploit the Intel profiling tools (inspector, vamplifier, etc...) with your Visual Studio and then tell to me (a real poor man without VS) how they works :-)

[zbeekman]

@szaghi I am “poorer” than @jacobwilliams: no VS for me.

BTW, @szaghi did you see my post on cpp-coveralls about what I think the issue is that you’re having? Any luck getting it to work?

[szaghi]

@zbeekman
I am sorry for my delay, but today was very busy. I just tested your suggestions on binutils and with a sym link of gcov into local path (that I export) cpp-coverals works very fine, thank you very much! Indeed, before I had to stop testing your suggestions, I had tried to use a container-based travis (as I guess you do) for exploiting travis caching feature, but I failing miserably (in particular installing gfortran-4.9 without sudo and without add the testing repo). Can I ask you how you use the travis container-based (if so)?

[zbeekman]

@szaghi no, we don’t use container based for this project due to the gfortran 4.9 dependency. There is no sane way, that I know of, to use a container based build and improve the build speed. An insane way would be to:

1. compile GCC/gfortran 4.9 from source on travis, without installing it to the restricted system directories
2. set up the environment to compile and run from from this new GCC build (that is in user space)
3. cache the built binaries so that you don’t have to re download and re-compile them next time travis is fired up

But I don’t particularly fancy scripting the installation of GCC from source on a remote VM. Which is why I don’t think that this is a sane approach. I just have cache: apt in .travis.yml in they ever add apt caching to the non container builds in the future.

[szaghi]

@zbeekman thank you. I agree with you, I do not like to locally compile gfortran. Thank you for your suggestions.

P.S. If I will find the time, I will try to support gprof in FoBiS and improve gcov one. Reading your nice idea to maintain a wiki page of untested procedures I have had the idea that this can be automatically accomplished by FoBiS: run the build with coverage flag should build the targets (your tests in this case) with coverage instruments, run the tests themself, collect the coverage data and produce a markdown page report. In FoBiS is simple to produce markdown pages ... Do you have suggestions?

[zbeekman]

P.S. If I will find the time, I will try to support gprof in FoBiS and improve gcov one. Reading your nice idea to maintain a wiki page of untested procedures I have had the idea that this can be automatically accomplished by FoBiS: run the build with coverage flag should build the targets (your tests in this case) with coverage instruments, run the tests themself, collect the coverage data and produce a markdown page report. In FoBiS is simple to produce markdown pages ... Do you have suggestions?

Hi Stefano,

I’m not sure what the current FoBiS.py support for gcov is beyond compiling/linking with coverage flags. I seem to recall you mentioning some additional support of some variety… if so can you remind me what the additional functionality does?

As for generating markdown pages for the coverage info… that is quite an interesting idea! I guess ideally it should be stylistically compatible-with/similar-looking-to with FORD since FoBiS.py and FORD use similar Markdown engines, yes? (Of course the ability to customize the appearance, for example, to be more similar to ROBODoc output, would be wonderful.)

As far as the actual output content goes, it would be great to:

• Ensure that extracted coverage information is updated each time a program runs. (i.e., the .gcda files are cumulative and the json-fortran test suite consists of a number of individual programs, so if FoBiS.py is going to call gcov for the user it must make sure that it does so either after all executables have been run, or for every source file of an executable after it’s run, each time it is run.)
• Include basic information at the top, for each source file like:
  • number of procedures (functions and subroutines)
  • number of executed procedures
  • number of unexecuted procedures
  • percentage of executed and unexecuted procedures (with pie chart?)
  • number of executable lines
  • number of missed lines
  • number of covered lines
  • average hits per total number of executable lines
  • average hits per executed line
  • pie chart of percentage of missed lines and percentage of hit lines?
• A functions listing sorted by percentage of missed lines—functions with high miss rates should be printed first
  • For each function include similar info as above:
    • number of executable lines
    • number of lines missed
    • number of lines hit
    • average hits per executable line
    • average hits per line actually executed
    • percentage of hits and misses, possibly with a pie chart

[szaghi]

Hi Izaak,

I try to summarize some points.

FoBiS.py coverage support

Presently, there are a coverage switch for both build and rule execution: the first build target with coverage instruments, while the second run the analyzer (gcovr for gnu and the intel equivaldnt for intel). However, the user have to run the targets into the middle of this two steps, and the rule step is very trivial (in my cases I do other things after... like coveralls as you). Thus I am now thinking (not yet an actual design) how to improve. In not special order I am thinking to:

• force cflags_heritage when coverage is activated in building mode: this ensures that all sources are always instrumented;
• running the targets after they have built;
• modify the intrinsic rule (the second coverage above) in order to make it more general (this is a very unclear stuff).

Unexecuted procedures

First of all FoBiS.py does not use any markdown inputs or outputs. Ford uses markdown for its input and jnija2 as template engine for its html outputs. I said that is simple for FoBiS.py to generate mardown reports because I have strong experience with Python+markdown due to another project of mine MaTiSSe. Consequently, I will very quickly integrate a markdown generator into FoBiS if necessary.

As uncovered procedures searching algorithm is concerned, I am very confused. Yesterday, with the fews minutes of my lunch break... I tried to extract uncovered informations by means of different tools, but it has been very frustrating:

• gcov & co. do not provide information about procedures, but only for each single line executed or not;
• gprof provide many informations (executed and not executed procedures list, timing, number of executions...) but it very unflexible:
  • I do not find a way to limit the informations provided to a specific set of files;
  • the calls graph is creasy: there are the calls to all, immaginable, procedures (libc system library, gfortran low level procedures...) and I do not find a way to trim out these low-level calls;
• after the failing of my approach with gprof I tried callgrind by means of vallgrind: it seems more friendly, but the results I obtained is essently the same of gprof.

Indeed my failing is surely due to the fews minutes I tested the tools, but I think I need your help to find a simple way to extract the uncovered procedures information:

• how do you extract your un-executed procedures list?
• how we can find if a procedure is a function or subroutine?
• robodoc parses markdown or you need other format (html, txt,...)?

Today I should find some more minutes to dig this depeer.

Thank you for help.

[zbeekman]

Hi Izaak,

I try to summarize some points.

FoBiS.py coverage support

Presently, there are a coverage switch for both build and rule execution: the first build target with coverage instruments, while the second run the analyzer (gcovr for gnu and the intel equivaldnt for intel). However, the user have to run the targets into the middle of this two steps, and the rule step is very trivial (in my cases I do other things after… like coveralls as you). Thus I am now thinking (not yet an actual design) how to improve. In not special order I am thinking to:

• force cflags_heritage when coverage is activated in building mode: this ensures that all sources are always instrumented;

yes, I think this is a good idea. I was confused and surprised a few times when coverage info wasn’t created after adding -coverage because the binaries had already been built and seemed to be up to date. Also, when I look at the contents of .cflags.heritage right now, it seems that -ftest-coverage and -fprofile-arcs aren’t in there… only the flags passed by the user to -cflags rather than all the flags that FoBiS.py passes to the compiler. I suspect this is a bug.

• running the targets after they have built;

I’m not sure what a good way to do this would be. Who knows if the targets will require special command line arguments or input files in a particular location?

• modify the intrinsic rule (the second coverage above) in order to make it more general (this is a very unclear stuff).

Unexecuted procedures

First of all FoBiS.py does not use any markdown inputs or outputs. Ford uses markdown for its input and jnija2 as template engine for its html outputs. I said that is simple for FoBiS.py to generate mardown reports because I have strong experience with Python+markdown due to another project of mine MaTiSSe. Consequently, I will very quickly integrate a markdown generator into FoBiS if necessary.

As uncovered procedures searching algorithm is concerned, I am very confused. Yesterday, with the fews minutes of my lunch break… I tried to extract uncovered informations by means of different tools, but it has been very frustrating:

• gcov & co. do not provide information about procedures, but only for each single line executed or not;

On my system there are two ways to get info about procedures. If you checkout the master branch of json-fortran and then export CODE_COVERAGE=yes and then ./build.sh, which will call FoBiS.py to do the build and then run the test suite. The code will be compiled with the gfortran coverage flags and the binary files containing the coverage info are cumulatively updated. Then here are the two ways to get the procedure coverage information:

1. Call gcov -o lib src/json_module.f90 to produce the json_module.f90.gcov file for the library. In it you will notice that lines preceded by #### are not executed. Any line that is a procedure declaration subroutine foo(in_a,in_b,out_1) that has the #### in front of it indicates that this procedure has never been called.
2. Another way is to pass the -f flag to gcov: gcov -f -o lib src/json_module.f90 This should produce the same json_module.f90.gcov file with the source listing and info about executed lines, BUT it will also print procedure execution summaries to stdout. An example of a procedure that is never called will look something like this:

Function '__json_module_MOD_json_file_move_pointer'
Lines executed:0.00% of 6

Note that the gcov man page says that percentages will never be listed as exactly 0 or 100 unless NONE or ALL of the lines are executed, respectively. So 0.00% always indicates no lines executed and hence the procedure was never called.

Either way, my suggestions will very likely be quite challenging to implement, since the output of gcov needs to be parsed to extract the relevant information. I think this is what cpp-coveralls does, and then encodes all this info in a json payload to send to coveralls.io.

FoBiS.py is a great tool, and anything beyond what you have already done is extra, in my mind. If you are up for this challenge, great, but if it’s too complicated, it doesn’t add that much compared to the other great functionality you’ve already implemented. I just thought I’d give you my “Christmas wish list” for what the ideal markdown output would look like.

• gprof provide many informations (executed and not executed procedures list, timing, number of executions…) but it very unflexible:
  • I do not find a way to limit the informations provided to a specific set of files;
  • the calls graph is creasy: there are the calls to all, immaginable, procedures (libc system library, gfortran low level procedures…) and I do not find a way to trim out these low-level calls;
• after the failing of my approach with gprof I tried call grind by means of vallgrind: it seems more friendly, but the results I obtained is essently the same of gprof.

I wouldn’t worry to much about gprof and callgrind. I think that extracting coverage information should be done with gcov or similar. Profiling info can be nice, but beyond sending gprof output through gprof2dot and then dot (part of graphviz) to generate images of the call graphs with timing info. It’s fine if system calls are included, in gprof2dot there is a way to set thresholds to not display calls that take insignificant time and this usually gets rid of a lot of the system calls. While “fortran poor man” may not entirely understand the system calls, the relevant info will still be there.

Indeed my failing is surely due to the fews minutes I tested the tools, but I think I need your help to find a simple way to extract the uncovered procedures information:

• how do you extract your un-executed procedures list?

As discussed above the best way is to parse the *.f90.gcov file for function and subroutine declarations preceded by #### (line never called). The -f, function summary, flag to gcov might also be useful.

• how we can find if a procedure is a function or subroutine?

I’m not sure whether or not this is critical. It should be clear to the user how to find the procedure in the source, since functions and subroutines cannot have the same name in the same scoping unit. However, if you parse the *.f90.gcov file, the line with the function or subroutine declaration, will include exactly that information.

• robodoc parses markdown or you need other format (html, txt,…)?

No, robodoc doesn’t know anything about markdown or do any parsing of markdown. What I meant was that it would be nice to be able to have output in a format that is stylistically compatible with other documentation, whether it is FORD or ROBODoc. The ability to have HTML output would also be great, then the robodoc.css can just be used as the stylesheet.

Today I should find some more minutes to dig this depeer.

Thank you for help.

I’m happy to help, but as I mentioned above all of these are bonus features. FoBiS.py is great as is! Only work on this if you are inspired to do so and have the time to spare.

[szaghi]

@zbeekman

Perfect!

I never take a look at the .gcov ouputs, I had wrongly supposed that there were only lines coverage without info about procedures...

Your Christmas wish list will be completed before the next Christmas... It should be very simple to parse the .gcov files. For our first step I will try to produce a very simple markdown report, in a second phase I will try to support other format.

Following your wise suggestions, I freeze (for the moment) any other experiments with gprof (indeed I have already added a -profile option...).

Tomorrow I hope to push a first FoBiS.py version with your -uncovered report feature :-)

P.S. the missing coverage flags into the cflags heritage is really a bug! Thank you very much!