Load is VERY slow for a NetCDF multi-variable file

[senesis]

📰 Custom Issue

When loading a single variable from a quite small NetCDF file which includes 300 variables, the load time is very large : around 100 seconds (while it is less than 0.1s for a similar single variable file).

This is a bottleneck for trying to use Iris (trough ESMValTool) for handling some climate model native data format.

The attached notebook load_time_histmth.pdf demonstrates the issue and includes a profiling, which shows that the most time consuming function is (by large) NetCDFDataProxy.getitem

The data file is available here

System info is :

uname -a
Linux ciclad-ng.private.ipsl.fr 2.6.32-754.35.1.el6.x86_64 #1 SMP Wed Oct 7 03:47:54 CDT 2020 x86_64 x86_64 x86_64 GNU/Linux

[bjlittle]

@senesis Thanks for taking the time to report this, much appreciated.

Could you just confirm the version of iris that you're using? I'm assuming v3.0.1?

[senesis]

Yes it is v3.0.1

[bjlittle]

@senesis Great thanks. And which version of Python? v3.8?

[bjlittle]

@senesis We have a patch in the pipeline that will go towards significantly alleviating this issue.

We're going to target this for the forthcoming iris v3.0.2 release 👍

See the v3.0.x Release Discussion for further details.

[pp-mo]

@senesis great issue -- your account is excellent + should make the problem reproducible !

I was already looking into issues with slow netcdf loads, specifically with lots of variables (i.e. we also found something similar).
I think I may already have a solution that will at least "alleviate" it ... coming shortly I hope.

In the meantime though, I wonder if you could put up your 'load_time_histmth' notebook so I can test out with that, maybe in a Gist ??

[pp-mo]

Stop press: see #4135
I just tested this with your file 'Iris_multivar_data_file.nc'
For me, it speeds up loading that (300-odd cubes) from ~50 to ~5 secs.
Win ! 😀

[senesis]

In the meantime though, I wonder if you could put up your 'load_time_histmth' notebook so I can test out with that, maybe in a Gist ??

I am not familiar with Gist. The notebook is available here

[senesis]

For me, it speeds up loading that (300-odd cubes) from ~50 to ~5 secs.

Great , I am looking forward to get 3.0.2 (and that ESMValTool uses it)