Performance problems with CastXML and py++

[iMichka]

The main effort was to make pygccxml and py++ work with CastXML. Now that this is done, one problem remains to be solved: performance. @mamoll reported that using py++ is now much slower than before.

When comparing the xml files (between gccxml and castxml), one thing which seems very different is for example the std namespace. All the internals are now exposed with CastXML. With gccxml std contained around 260 elements. With castxml it has more than 20000 … This is probably making pygccxml slow when parsing for declarations.

This will need to be benchmarked and tested. Probably we can just skip some of these declarations or re-order the dictionaries / lists to make the algorithm faster. I plan to do this for the 1.8.0 release.

[praetorian20]

I just ran into this a couple of days ago. A huge header that took about 90s to generate bindings in -std=c++03 mode now takes 60m with -std=c++14. I've been tweaking things in pygccxml and have managed to get the time down to 25m (which is still not acceptable). I'll submit a PR with the changes once I get it down some more.

What was surprising was I saved about 10m by not calling scopedef_t.init_optimizer (which is supposed to optimize later queries). @mamoll If you get a chance, could you please try the same to verify you see a speed up too? To do this, add optimize_queries=False when you're calling the module_builder.module_builder_t constructor in pyplusplus.

[iMichka]

Hi. Thanks for reporting. I was also playing around with a profiler. Made already some slight changes.
I know about init_optimizer; this is one of the first things that struck me when profiling.

I do not want to disable it completely. I think that we could disable the optimisers on the files that are not the files that are parsed (by looking at they location). But this is only the tip of the iceberg: there are multiple other places where pygccxml gets slow because of the number of declarations. Looping over arrays or dictionaries with more than 20000 elements, and that hundreds of time, si very slow.

We may need to have two declarations trees, one hight priority and the other one slow priority, to optimise lookup times. But this is an huge amount of work. Maybe there would be another nicer strategy, I am still thinking about this and experimenting.

This is just to let you know that I am also working on this. I may just allow to disable the optimisers optionally for 1.8.0, bring that release out, and work on real performance improvements later.

[mamoll]

Adding optimize_queries=False seems to have only a small effect in my case. The python binding runtime for my project is still "a good part of a day" with gcc5 (whereas with clang it's "a bit more than an hour").

[iMichka]

Looking at the Travis build (I know, this is not a perfect metric, but looking at the mean build time over multiple builds is not that bad), the test suite takes between 3 to 4 min to complete with gccxml. Using castxml with std=c++98 and gcc 4, it's almost the same. The builds that take longer are the ones using std=c++11; and even longer std=c++14 (with gcc5) (8 to 9 minutes).

On my laptop, the test suite goes from 198s to 252s when using std=c++14 with castxml.

At first I was trying to optimise for std=c++98, because I did not think about it. I will try to compare the time of each test, to see which are the ones which are taking the most time. This will hopefully allow to refine the testing process and setup a test case demonstrating the performance problems.

[praetorian20]

@mamoll Thanks for trying that out. What standard library implementation are you using with clang, libc++ or libstdc++? I'm curious whether this is a case of gcc being slow or if it's differences between the stdlibs.

@iMichka I'm happy to hear you're looking into this too. I haven't started rewriting any logic, all I've done so far is cached the results of some functions that were called repeatedly, and eliminated a few checks that were happening on each function call in some places.

It's likely further improvements will require some rewrites, so I'll make a PR out of what I have so far later today or tomorrow.

Not sure I understand what you mean about init_optimizer, I don't think you need to allow disabling it within pygccxml because it doesn't get called automatically (as far as I know). In my case, it was being called from pyplusplus, and I was able to disable that using the argument I mentioned above.

[mamoll]

@praetorian20 I am comparing clang + libc++ in the latest Xcode on El Cap with gcc5 + libstdc++ on Linux VMs. You could be right that the C++ std lib is the main culprit, not the compiler itself.

[iMichka]

@praetorian20 What I meant about the init_optimizer part is that we could probably be smart about when to use it: it should probably run on the declarations from the file you are parsing but not on the declarations defined in some std library files. Though I do not know if that is a good strategy. I am still worried that so many declarations are exposed. We may be able to optimise some bits, but at the end some methods still get called millions of times, which is expensive.

Here is a list of tests that run slower (in seconds). The comparison is between castxml + gcc6 + std=c++98 and castxml + gcc6 + std=c++14. I took all the tests that run for 1 second longer with std=c++14. (The list is not sorted)

I am currently profiling these to see if any speed gain is possible.

declaration_files_tester.Test.test 1.42260813713
declarations_comparison_tester.Test.test_comparison_declaration_by_declaration 25.1094276905
declarations_comparison_tester.Test.test_comparison_from_reverse 8.47017979618
declarations_comparison_tester.Test.test_same_declarations_different_intances 8.55400776866
file_cache_tester.Test.test_from_file 8.92868590356
file_cache_tester.Test.test_update 2.37047886848
project_reader_correctness_tester.Test.test_correctness 17.9011819363
type_traits_tester.Test.test_has_any_non_copyconstructor 3.31692194939
core_tester.core_all_at_once_t.test_abstract_classes 1.57977294922
xmlfile_reader_tester.Test.test 11.0374550819
complex_types_tester.Test.test 1.91197180748
cached_source_file_tester.Test.test 2.7938029766
variable_matcher_tester.tester_2_t.test_no_defaults 2.90506100654
decl_printer_tester.Test.test_printer 3.15186810493
vector_traits_tester.Test.test_declaration 1.98915410042
dependencies_tester.Test.test_calldefs 2.78062915802
remove_template_defaults_tester.Test.test_deque 3.28198480606
find_container_traits_tester.Test.test_find_traits 11.1609299183
better_templates_matcher_tester.Test.test 2.21111297607
from_future_import_tester.Test.test_namespace_argument_int 2.53657603263

[iMichka]

I cached a bunch of things, and removed a function layer that added much overhead. There is a clear performance improvement. Running the test suite went down from 252s to 141s on my laptop, and some travis builders are now faster.

Of course this does not mean that pyplusplus will be faster, I did not test this in a real-life context.

Still, the performance is not as good as when using gccxml or using the std=c++98 flag. There is still room for improvement.

• I can cache more things; but I am afraid that RAM usage will increase a lot.
• Each declaration has it's own cache. I think it would be more performant to have a single cache object for all the declarations. Maybe I could write something like an LRU cache, but I need to benchmark that first. Not sure if this is the right caching strategy (maybe there are others that may work better here).
• Of course more profiling needs to be done, but it is very time-consuming to find out what is going on.

Another important thing: I just read a comment in the pyplusplus code:

  To speed up your generation process you might want to consider making the
  declaration tree as small as possible and only store those declarations that
  somehow have an influence on the bindings. Ideally, this is done as early
  as possible and luckily gccxml provides an option that allows you to reduce
  the number of declarations that it will store in the output XML file. You can
  specify one or more declarations using the ``-fxml-start`` option and only
  those sub-tree starting at the specified declarations will be written. For
  example, if you specify the name of a particular class, only this class
  and all its members will get written. Or if your project already uses
  a dedicated namespace you can simply use this namespace as a starting point
  and all declarations stemming from system headers will be ignored (except
  for those declarations that are actually used within your library).

  In the ``pygccxml`` package you can set the value for the ``-fxml-start``
  option using the ``start_with_declarations`` attribute of the
  ``pygccxml.parser.config_t`` object that you are passing to the parser.

This option also works for castxml. I think ompl is not using the start_with_declarations option. This will definitively speed things up. Could you test this ?

[mamoll]

The start_with_declarations option speeds things up significantly! Module generation time went down from 3-4 hours per module to about 30 minutes per module. I need to do some more testing to see if there are any significant differences in generated code, but this seems extremely promising. I used pygccxml 1.7.6, so the timing results do not include your other performance improvements yet. Doh, I ran this test on OS X with clang, where binding generation time was already on the order of half an hour per module (i.e., using start_with_declarations made no difference). I am rerunning the test now on a Linux VM where it takes hours per module.