misc. benchmark regressions since 0.4

[JeffBezanson]

Just ran make test-perf and observed this. Most results are the same, and we've improved on about 6 tests, but unfortunately there are some significant regressions:

test	0.4	4f65737	4f65737 -O3	b029f50
parse_int	0.190	0.315		0.245
cons	80.837	91.86		90.14
gk	52.035	62.92		63.84
sparsemul	32.5			37.14 (threads)
sparsemul2	62.836	65.73	63.23	68.5
simplex	39.209	55.02	39.90	43.83
splitline	49.86	56.87	60.86	67.44
add1	33.63	44.09	37.82	36.6
add1_logical	41.797	105.3	60.12	45.4
binary_trees	25.5			29.64
k_nucleotide	61			75.34 (threads)

[JeffBezanson]

There is also a significant regression in the spell benchmark.

0.4:

spell              10090.812 10533.558 10285.806  183.614

0.5:

spell              11888.845 12585.925 12231.564  297.824

I tried changing some of the containers to use String instead of AbstractString but it didn't seem to help much. Using SubString{String} as the key type of the Dict in train gives

spell              11395.126 11790.546 11550.910  169.565

slightly better but still slower than 0.4 and still 2.5x slower than python.

[StefanKarpinski]

Some of the planned changes may improve this – in particular, the new Char type.

[tkelman]

We should probably try to track down what led to the slowdown though - then any internal representation changes would be speedups, not just countering regressions elsewhere.

[hayd]

a retrospective daily (?) history of the perf tests might help, that way it could be seen whether it was single/few commits which were to blame. cc @jrevels

[JeffBezanson]

Looks like #16236 improved things a bit, but not enough.

[JeffBezanson]

Spell test after #16236:

spell              11070.436 11496.856 11296.590  204.591

[StefanKarpinski]

It probably makes sense to run benchmarks on the version right before the String change to figure out how much of this is due to various other changes and how much is due to that change.

[jrevels]

It probably makes sense to run benchmarks on the version right before the String change to figure out how much of this is due to various other changes and how much is due to that change.

See here. Hopefully I got the commit range correct - I meant to do the most recent commit related to #16058 vs. the commit right before #16058 in the history.

[vtjnash]

I wonder how much of this can be traced back to #15997. I was trying to track down a recent performance regression of #15541 (thinking I had broken typemap), and found that @tkelman commit 5366586 was responsible for the ~15-20% performance regression in that test.