PI-3473: Netcdf loading ancillary variables #3556

pp-mo · 2019-11-20T17:06:28Z

Load ancils for netcdf.
STATUS: why this targets the 'launch_ancils' feature branch
We aren't yet sure that we will be ready to "release" ancillary variables in Iris 3.0
(mostly because there are a lot of places where specific handling is needed to avoid creating bad cubes).
Hence this is the key PR on the launch_ancils feature branch.

In practice, some support has already been merged to master.
Especially #3422, which also already makes the cube API for ancils visible.
However.. bad things don't happen until cubes contain ancils, which will generally only happen to users when we merge this PR, following which they can be generated when loading.
Likewise, we are also "holding back" on other PRs, targetting the feature branch instead of master), e.g. documentation additions

NOTES:

also silently fixes an hitherto unknown problem, whereby attributes of cell-measures were not loaded.
- possibly there should be an extra, specific test for that ? But I think it is not really needed.
only one test, at present

stephenworsley · 2019-11-26T14:09:05Z

lib/iris/tests/test_netcdf.py


        self.assertCML(cubes, ("netcdf", "netcdf_cell_methods.cml"))

+    def test_ancillary_variables(self):


It might also be worthwhile also testing for cell measures. As far as I can tell, they aren't currently being tested for.

…n code (SciTools#3571)

stephenworsley

Looks good, I've convinced myself this treats cell measures and ancillaries in a way that is consistent with current treatment of coordinates. Just a couple very minor things to look at for tidiness sake.

stephenworsley · 2019-12-04T11:01:24Z

lib/iris/fileformats/_pyke_rules/fc_rules_cf.krb

+        attr_units = get_attr_units(cf_cm_var, attributes)

-        data = _get_cf_var_data(cf_cm_attr, engine.filename)
+        data = _get_cf_var_data(cf_cm_var, engine.filename)


For consistency, should this have a comment similar to this line?

iris/lib/iris/fileformats/_pyke_rules/fc_rules_cf.krb

Lines 2018 to 2019 in 1178de7

# Get (lazy) content array

data = _get_cf_var_data(cf_av_var, engine.filename)

Ok, but "consistency" is a bit of a lost cause in this code IMHO. Not least because we have far too much code duplication code, as here !
( N.B. We do still hope to replace all of this with Python code #3415 )

stephenworsley · 2019-12-04T11:09:21Z

lib/iris/fileformats/netcdf.py

        coord.var_name: coord.standard_name or coord.var_name or "unknown"
        for coord in cube.coords()
    }
-


Should there still be whitespace here?

Ok, I suppose so.
I might have hoped that Black would take care of this, but clearly the results are not totally standardised ..

* Revert to single-line command for doctr invocation. * Added script comment, partly to force Github respin.

pp-mo · 2019-12-09T17:11:39Z

Thanks @stephenworsley
Hopefully addressed your previous comments now.
Please re-review.

stephenworsley

Looking good, just one more small change needed.

stephenworsley · 2019-12-11T10:32:10Z

lib/iris/tests/test_netcdf.py

+                    my_areas:custom = "extra-attribute";
+            data:
+                axv = 11, 12, 13;
+                ayv = 21, 22, 23;


ayv should be length 2.

Thanks.
I'm surprised it didn't error that!

pp-mo · 2019-12-12T16:48:48Z

Hi @stephenworsley
Fixed ?

stephenworsley · 2019-12-12T17:19:10Z

@pp-mo Looks good. I think this just needs rebasing to make Travis pass

stephenworsley

Looks good, ready to merge!

* _regrid_area_weighted_array: Tweak variable order to near other use in code (SciTools#3571) * Fix problems with export and echo command. (SciTools#3577) * Pushdocs fix2 (SciTools#3580) * Revert to single-line command for doctr invocation. * Added script comment, partly to force Github respin. * Added whatsnew for Black. (SciTools#3581) * Fixes required due to the release of iris-grib v0.15.0 (SciTools#3582) * Fix python-eccodes pin in travis (SciTools#3593) * Netcdf load of ancillary vars: first working.

… launch_ancils * 'launch_ancils' of https://github.com/SciTools/iris: Resolve conflicts with requirements/test.txt to update the imagehash pin to imagehash>=4.0 (SciTools#3735) PI:3358 (WIP) Ensure flags load/save without units (SciTools#3613) PI-3473 Rename "engine.provides" (SciTools#3590) PI-3473: Whatsnews relating to ancillary load + save. (SciTools#3557) PI-3473: Netcdf loading ancillary variables (SciTools#3556) � Conflicts: � lib/iris/fileformats/netcdf.py � lib/iris/tests/unit/fileformats/netcdf/test__load_aux_factory.py � requirements/core.txt

pp-mo added this to the v3.0.0 milestone Nov 20, 2019

lbdreyer mentioned this pull request Nov 20, 2019

Add Support for Loading/Saving Ancillary Variable to/from NetCDF #3473

Closed

3 tasks

pp-mo mentioned this pull request Nov 21, 2019

PI-3473: Whatsnews relating to ancillary load + save. #3557

Merged

abooton changed the title ~~Netcdf loading ancillary variables~~ P.3473: Netcdf loading ancillary variables Nov 26, 2019

abooton changed the title ~~P.3473: Netcdf loading ancillary variables~~ P3473: Netcdf loading ancillary variables Nov 26, 2019

abooton changed the title ~~P3473: Netcdf loading ancillary variables~~ PI3473: Netcdf loading ancillary variables Nov 26, 2019

abooton changed the title ~~PI3473: Netcdf loading ancillary variables~~ PI-3473: Netcdf loading ancillary variables Nov 26, 2019

stephenworsley reviewed Nov 26, 2019

View reviewed changes

pp-mo changed the base branch from master to launch_ancils November 29, 2019 16:39

pp-mo mentioned this pull request Nov 29, 2019

PI-3483 Ensure documentation of ancillary variables handling #3570

Closed

trexfeathers mentioned this pull request Nov 29, 2019

Ancillary Variables - validate 'all other' cube operations #3554

Closed

7 tasks

_regrid_area_weighted_array: Tweak variable order to near other use i…

6842a50

…n code (SciTools#3571)

stephenworsley requested changes Dec 4, 2019

View reviewed changes

stephenworsley self-assigned this Dec 4, 2019

pp-mo and others added 4 commits December 4, 2019 13:28

Fix problems with export and echo command. (SciTools#3577)

5cfec8f

Pushdocs fix2 (SciTools#3580)

26a32a3

* Revert to single-line command for doctr invocation. * Added script comment, partly to force Github respin.

Added whatsnew for Black. (SciTools#3581)

242a62f

Fixes required due to the release of iris-grib v0.15.0 (SciTools#3582)

479b2cf

pp-mo mentioned this pull request Dec 9, 2019

PI-3473 Rename "engine.provides" #3590

Merged

stephenworsley requested changes Dec 11, 2019

View reviewed changes

Fix python-eccodes pin in travis (SciTools#3593)

738093f

pp-mo added 3 commits December 12, 2019 17:42

Netcdf load of ancillary vars: first working.

365d0e9

Review changes.

c683f2e

Review change.

5c50c56

pp-mo force-pushed the ancil_load branch from ab0ac80 to 5c50c56 Compare December 12, 2019 17:42

stephenworsley approved these changes Dec 13, 2019

View reviewed changes

stephenworsley merged commit 7703699 into SciTools:launch_ancils Dec 13, 2019

abooton mentioned this pull request Dec 20, 2019

PI-3473 Rename "engine.provides" (#3590) #3622

Closed

abooton mentioned this pull request Jun 10, 2020

[FB] Launch ancils: reset mergebase to commit 738093f #3733

Merged

pp-mo deleted the ancil_load branch March 18, 2022 15:27


		self.assertCML(cubes, ("netcdf", "netcdf_cell_methods.cml"))

		def test_ancillary_variables(self):

	# Get (lazy) content array
	data = _get_cf_var_data(cf_av_var, engine.filename)

PI-3473: Netcdf loading ancillary variables #3556

PI-3473: Netcdf loading ancillary variables #3556

Uh oh!

Conversation

pp-mo commented Nov 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stephenworsley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pp-mo commented Dec 9, 2019

Uh oh!

stephenworsley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pp-mo commented Dec 12, 2019

Uh oh!

stephenworsley commented Dec 12, 2019

Uh oh!

stephenworsley left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pp-mo commented Nov 20, 2019 •

edited

Loading