NetCDF loading time breakdown

[bblay]

Time the various processes used in NetCDF loading, to inform the decision on where and how to improve the performance.

For example, the performace problem is described here.

If the results clearly show the way forward, create a new issue for the work package(s).

[rhattersley]

Using one particular user-supplied test, the biggest impact on performance was from repeated creation of large AuxCoords (~100MB of coordinate data loaded for each 5MB of phenomenon data!). See #216 for the fix.

After that, netCDF attribute access was singled out as a hotspot - see #213.

[rhattersley]

Also submitted #218 to speed up the creation of masked arrays.

[rhattersley]

Progress summary: a job (load data & compute area weighted sum) which initially took several minutes to run now takes five or six seconds. Substantial progress! 😄

[bblay]

😮

[rsignell-usgs]

I see this issue is closed, but with the latest release (1.5 dev), it still takes a very long time to get a cube list on a dataset that has many variables with auxiliary coordinates. For example, this load command takes over 5 minutes:

url = 'http://geoport.whoi.edu/thredds/dodsC/coawst_4/use/fmrc/coawst_4_use_best.ncd'
cubes = iris.load(url)

I looked at the OPeNDAP server log, and this load command issues 1339 data requests to the server, requesting the same coordinate variables over and over (e.g., it requests the 2D lon/lat arrays aux coordinate lon_rho and lat_rho 69 times each).

Could there be a way to avoid rereading coordinates that have already been read, or delay reading of these coordinates?

Needless to say, I've been avoiding this dataset in Iris demos (but that's unfortunate, since this is the only Forecast model we run here at USGS). ;-(

[pelson]

Could there be a way to avoid rereading coordinates that have already been read, or delay reading of these coordinates?

Yes - that is definitely on the cards. We've got biggus waiting in the wings, which just needs a little push to get it up to standard, and then we can delete large amounts of Iris code and gain easy deferred loading of coordinate variables (auxiliary definitely, and potentially dimension coordinates too...)

[rsignell-usgs]

Excellent. I'd like to be on the waiting list for alpha or beta testing! ;-)