Compare tests timings (3 man days' effort)

[pp-mo]

Before+ after dask refactoring (or whatever changes)
STATUS:
not sure this is quite done, as results are a bit meaningless -- see comment below

[pp-mo]

Code now up for doing these measurements : #2372

Will from time to time do measures + post results here as attached files.
Generally for now, will use machine eld118.
For method, see usage note in iris/tests/__init__.py

[pp-mo]

Preliminary tests, using eld238 and three branches :

• test_timings (as Test timings #2372 )
• dask_w_timings = dask, at this commit, rebased onto test_timings)
• dask_cube_speed_timings = above, merged with dask_cube_speed

Note: "dask_cube_speed" is the fix here : #2376

Raw data enclosed:

• Notes.txt
• iris_testoutput_20170214_br_test_timings.txt
• iris_testoutput_20170214_br_dask_w_timings.txt
• iris_testoutput_20170215_br_dask_cube_speed_timings.txt

NOTE: not quite done with this..
I'll produce some code to analyse these raw results.

[pp-mo]

STATUS:
not sure this is quite done, as results are a bit meaningless
We need to be able to

• chop out the relevant data + produce a table-like representation
• compare timings for same test across different result sets
• find which tests show significant differences in timing
• ignore really short ones

[marqh]

the data collection test harness is merged to master now (#2372)

the data collection and data processing can be addressed based on commits. We can rebase the dask branch onto master to get all of these.

I think this ticket should either be 'in progress' waiting for that to happen, or closed and we'll analyse statistics later

@pp-mo do you have a preference?

[pp-mo]

I think we should finish getting these changes onto the feature branch first,

However we currently have a problem in that I rebased "dask" onto (my) "test_timings", and the result is stack-dumping when tested, at least with timings on.
I will chase this.

We should also at some point rebase "dask" onto master.
Now is probably better than later, but we need to decide if that makes sense.
I didn't do that initially for fear that other changes might influence test timings.

[pp-mo]

result is stack-dumping when tested
Proposed solution here : #2395

NOTE: it looks like the multiprocessing nosetests implementation was just ignoring the failure here : the problems don't show up on Travis.

[marqh]

I think we should finish getting these changes onto the feature branch first,

this is now done, although the result is that the dask feature branch is now called dask_timed