Wrong error message for duplicate coordinate

[schlunma]

The error message which appears in #371 is:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable tas:
height2m: does not exist
in cube:
air_temperature / (K)               (time: 1200; latitude: 180; longitude: 288)
     Dimension coordinates:
          time                           x               -               -
          latitude                       -               x               -
          longitude                      -               -               x
     Scalar coordinates:
          height: 2.0 m
             description='~2 m standard surface air temperature and surface humidity  height'
          height: 2.0 m
     Attributes:
          Conventions: CF-1.7 CMIP-6.0 UGRID-1.0
          activity_id: CMIP
          branch_method: standard
          branch_time_in_child: -674885.0
          branch_time_in_parent: 0.0
          comment: <null ref>
          contact: gfdl.climate.model.info@noaa.gov
          data_specs_version: 01.00.27
          experiment: pre-industrial control
          experiment_id: piControl
          external_variables: areacella
          forcing_index: 1
          frequency: monC
          further_info_url: https://furtherinfo.es-doc.org/CMIP6.NOAA-GFDL.GFDL-CM4.piControl.none...
          grid: atmos data regridded from Cubed-sphere (c96) to 180,288; interpolation...
          grid_label: gr1
          initialization_index: 1
          institution: National Oceanic and Atmospheric Administration, Geophysical Fluid Dynamics...
          institution_id: NOAA-GFDL
          interp_method: conserve_order2
          license: CMIP6 model data produced by NOAA-GFDL is licensed under a Creative Commons...
          mip_era: CMIP6
          nominal_resolution: 100 km
          original_name: tas
          parent_activity_id: CMIP
          parent_experiment_id: piControl-spinup
          parent_mip_era: CMIP6
          parent_source_id: GFDL-CM4
          parent_time_units: days since 1850-1-1 00:00:00
          parent_variant_label: r1i1p1f1
          physics_index: 1
          product: model-output
          realization_index: 1
          realm: atmos
          references: see further_info_url attribute
          source: GFDL-CM4 (2018): 
aerosol: interactive
atmos: GFDL-AM4.0.1 (Cubed-sphere...
          source_file: /mnt/lustre02/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ/CMIP/NOAA-GFDL/G...
          source_id: GFDL-CM4
          source_type: AOGCM
          sub_experiment: none
          sub_experiment_id: none
          table_id: Amon
          title: NOAA GFDL GFDL-CM4 model output prepared for CMIP6 pre-industrial cont...
          variable_id: tas
          variant_info: N/A
          variant_label: r1i1p1f1
     Cell methods:
          mean: area, time

However, the coordinate height is not missing, but appears twice (because it was additionally added by the fix). This can be very confusing.

The reason for this is that iris also raises an CoordinateNotFoundError for cube.coord('coord_name') if 'coord_name' appears more than once in cube:

iris.exceptions.CoordinateNotFoundError: 'Expected to find exactly 1 coordinate, but found 2. They were: ..., ....'

[valeriupredoi]

but it's impossible to add the same coordinate twice in a cube - even if one is DimCoord and the other is AuxCoord, no?

[schlunma]

Apparently it's possible if the two coordinates are slightly different, e.g. if one has additional attributes and the other one doesn't, like it's the case for the example posted above (one height coordinate has a description, the other one doesn't).

[valeriupredoi]

Pff, smells like an iris bug to me then Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Wed, 4 Dec 2019, 09:05 Manuel Schlund, ***@***.***> wrote: Apparently it's possible if the two coordinates are slightly different, e.g. if one has additional attributes and the other one doesn't, like it's the case for the example posted above (one height coordinate has a description, the other one doesn't). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#373?email_source=notifications&email_token=AG5EFI23QH54OHL74P5E6F3QW5XGZA5CNFSM4JLTRLK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF4H3AA#issuecomment-561544576>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5EFI6H4CWJNQWJK3IYPGLQW5XGZANCNFSM4JLTRLKQ> .

[schlunma]

I don't think so. Why would they add an error message like this then?

iris.exceptions.CoordinateNotFoundError: 'Expected to find exactly 1 coordinate, but found 2. They were: ..., ....'

[valeriupredoi]

so the error message takes care of the already created problem; the problem should not happen in the first place ie adding two longitude's as both DimCoords should not be allowed methinks

[schlunma]

Adding two DimCoords for the same index is not possible anyway, but height is an AuxCoord

[valeriupredoi]

yeah but what happens if the data is 3D and you remove say, time and add longitude in the place of time (so index 0), with the data now having longitude-longitude-latitude coords -> will that work? Coz if it does, it's bonkers 🍺

[schlunma]

It works if the metadata of the DimCoord is different (changing the points is not enough):

Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import iris
>>> d1 = iris.coords.DimCoord([1, 2], standard_name='latitude')
>>> d2 = d1.copy()
>>> d3 = d1.copy([2, 4])
>>> d4 = iris.coords.DimCoord([1, 2], standard_name='latitude', var_name='lat')
>>> cube = iris.cube.Cube([[1, 2], [3, 4]])
>>> print(cube)
unknown / (unknown)                 (-- : 2; -- : 2)
>>> cube.add_dim_coord(d1, 0)
>>> cube.add_dim_coord(d1, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/mlr/lib/python3.7/site-packages/iris/cube.py", line 1054, in add_dim_coord
    raise ValueError('The coordinate already exists on the cube. '
ValueError: The coordinate already exists on the cube. Duplicate coordinates are not permitted.
>>> cube.add_dim_coord(d1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/mlr/lib/python3.7/site-packages/iris/cube.py", line 1054, in add_dim_coord
    raise ValueError('The coordinate already exists on the cube. '
ValueError: The coordinate already exists on the cube. Duplicate coordinates are not permitted.
>>> cube.add_dim_coord(d2, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/mlr/lib/python3.7/site-packages/iris/cube.py", line 1054, in add_dim_coord
    raise ValueError('The coordinate already exists on the cube. '
ValueError: The coordinate already exists on the cube. Duplicate coordinates are not permitted.
>>> cube.add_dim_coord(d3, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/mlr/lib/python3.7/site-packages/iris/cube.py", line 1054, in add_dim_coord
    raise ValueError('The coordinate already exists on the cube. '
ValueError: The coordinate already exists on the cube. Duplicate coordinates are not permitted.
>>> cube.add_dim_coord(d4, 1)
>>> cube
<iris 'Cube' of unknown / (unknown) (latitude: 2; latitude: 2)>
>>> 

[valeriupredoi]

Bahahah, you got a Frankencube 🤣 I think we should probably tell the iris guys about this - ping @bjlittle

[bjlittle]

@schlunma and @valeriupredoi

An interesting question, and quite topical... see SciTools/iris#3583 which is part of an overhaul of iris cube-arithmetic that I'm doing at the moment (also see SciTools/iris#3478) and aiming to land in the forthcoming 3.0.0.

This all comes down to the question of "What is the identity of a coordinate?".

To be fair we need to communicate this more clearly, but essentially the answer comes down to the metadata of the coordinate, and in this case that is a combination of the coordinate:

• standard_name
• long_name
• var_name
• units
• attributes
• coord_system
• climatological (new in 2.3.0 and 3.0.0)

The identify of a coordinate has nothing to do with the value of its points and/or bounds. It's all about the associated metadata.

In your example, only the DimCoords d1 and d4 (has a var_name) differ in metadata, therefore their identity is different. Whereas, d1, d2 (exact copy of d1), and d3 (differs only in points, which is not part of the metadata, and therefore not part of the coordinate identity contract) are in metadata terms all the same - therefore that is why iris raises the exception.

Hope this helps to clarify.

I want to make this a slicker (and configurable) experience for the user and SciTools/iris#3583 is the first step in that direction - plus we need to clearly communicate all of this in the user guide to avoid any on-going confusion.

So IMHO this isn't a bug 😃

[rcomer]

For info, I have previously queried the Frankencubes: SciTools/iris#2917

[bjlittle]

"Frankencubes" -- it's now officially a thing 🤣

I've been thinking about Frankencubes over the weekend, and given that I'm kinda actively working in this area of iris at the moment, I may be able to address this behaviour in time for 3.0.0. No promises though...

I stand by my last comment that this isn't a bug, but then again, the double height coordinate in the cube behaviour is surprising for users and takes some explaining to justify... which hints to me that perhaps a less surprising compromise could be had.

Ideally, what you might want to happen here is for iris to raise an exception to say that the height coordinate already exists in that cube instance... and that comes down to relaxing the contract for coordinate metadata equality. I think there is a common sense compromise to be had here, that's not too upsetting for all concerned...I just need to figure that out and see if it make sense.

So, if you're happy, leave this issue open, and let's see if I can get the Frankencube to learn some table manners 😄

That said, your temporary workaround here might be something along the lines of:

height = iris.coords.DimCoord(2.0, standard_name="height", units="m", attributes=dict(description="...")
coords = cube.coords(height.name())
if not coords:
    # This isn't a Frankencube...
    cube.add_aux_coord(height)
else:
    # A "height-like" coordinate already exists in the cube...
    if len(coords)>1:
        raise ValueError("oops")
    # Add the extra metadata (what ever it is) to the existing "height" coordinate on the cube...
    coords[0].attributes.update(height.attributes)

[schlunma]

That sounds great @bjlittle, thanks for your help!!