Why are ScalarCoords listed in cube.aux_coords?

[kaedonkers]

🐛 Bug Report

The tuple that is cube.aux_coords contains scalar coords. Why? Surely ScalarCoords are not the same as AuxCoords? This presents a problem when trying to remove AuxCoords from a cube, for instance to deal with errors when using cube.coords(dimensions=[0]) (see #3324).

How To Reproduce

Inspect cube.aux_coords for a cube containing both AuxCoords and ScalarCoords. You will find that both types of coords are present in the tuple.

Expected behaviour

cube.aux_coords returns a tuple containing only AuxCoords.

Environment

• OS & Version: RHEL7
• Iris Version: 3.1.0

[pp-mo]

In Iris terms, scalar coords are aux-coords :
That is, they are the among the coords of a cube which are not dimension coords, or derived coords.
If you build a cube directly with Cube(dim_coords_and_dims=xx, aux_coords_and_dims=yy), they are included with the other aux-coords.

They are listed separately from Auxiliary Coordinates in a cube printout, but that is really just for visual convenience :
Internally, that is in terms of how they are stored, they are aux-coords, but just ones which have no cube dims.
https://github.com/SciTools/iris/blob/main/lib/iris/_representation/cube_summary.py#L251-L255

Possibly a confusion is due to dimensionality/shape :
For technical reasons, although they don't map any cube dimension, coord.cube_dims() == (), they still always have a shape coord.shape == (1,) and coord.ndim == 1, rather than shape=() and ndim=0 as you might expect.
I seem to recall there was a good technical reason for this ... but not what it actually was !! Probably something to do with numpy ??

[pp-mo]

Also in CF terms...
They are auxiliary coordinates because they are attached to a data variable via its 'coordinates' attribute
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#scalar-coordinate-variables
This is what distinguishes auxiliary coordinates from dimension coords : Dimension coords are attached (implicitly) simply by the data variable mapping the dimension.

[pp-mo]

Sorry for all this noise, but....
@kaedonkers I forgot to ask what was the actual problem here (!)
-- you need to deal with scalar coords as a separate sub-class in some way?
Obviously you can get that info, but what do you think is missing ?

[kaedonkers]

Thanks @pp-mo! I was not aware that scalar coords are categorised as auxillary coords. I guess the confusion for me was based on behaviour of a cube when subsetting.

If I am subsetting a cube such that one of the dim coords dimensionality is reduced to 1 (i.e. slicing), then that dim coord gets demoted to a scalar coord. The confusion starts when I then called cube.aux_coords and those same scalar coords were present in the aux_coords tuple as DimCoords. I am guessing those scalar coords were DimCoord objects since they were derived from DimCoords and their type was not changed in the process. Hence my concern that something was awry in the cube.aux_coords code.

I see three potential solutions:

1. I start reading CF conventions before bed and stop assuming I know how Iris works.
2. If scalar coords are truely aux coords, then their object type should change as a result
3. cube.aux_coords contains only aux coords

[pp-mo]

I am guessing those scalar coords were DimCoord objects since they were derived from DimCoords and their type was not changed in the process.

This is a classic gotcha in Iris ..
The class of a Coord (DimCoord or AuxCoord) in Iris does not match whether is a "dimension coordinate" or "auxiliary coordinate". So ...

• dimension coordinates must be a DimCoord
• but auxiliary coordinates can be either DimCoord or AuxCoord
• frequently (e.g. in loading), a coord will be created as a DimCoord "if it can be", and an AuxCoord otherwise
  • .. and since "if it can be a DimCoord" generally means "if the points are monotonic", that is always true for scalar coords

What this basically comes down to is that Iris "should have" chosen more useful class names, but unfortunately its a really ancient decision and unlikely now ever to be fixed ...

As a related awkwardness, Iris also does not use the same terms as CF for types of coordinate

• Iris "auxiliary coordinate" == CF "auxiliary coordinate"
• Iris "dimension coordinate" == CF "coordinate"

But I would say that is largely a 'flaw' in CF, as it doesn't have a term for "coordinate or auxiliary coordinate", and it actually makes inventing such a term rather awkward. But again, presumably, an ancient decision that we all now have to live with.

I've long thought this would be worth explaining "somewhere" in the Iris docs, but it really needs a suitable place to live
-- probably not in the UserGuide, maybe a further-topics "Iris and CF" section ?
(plus, I think 'further topics' and 'technical papers' need merging)
Do you have any bright ideas on improving this @kaedonkers ?

[pp-mo]

three potential solutions:
I start reading CF conventions before bed and stop assuming I know how Iris works.

Unfortunately, I think this is a case of "start reading the Iris docs ... and stop expecting it to behave as you expected from reading CF" 😢

[pp-mo]

IN the specific case...

If I am subsetting a cube such that one of the dim coords dimensionality is reduced to 1 (i.e. slicing), then that dim coord gets demoted to a scalar coord.

I gues this should read " If I am subsetting a cube such that one of the dimension coords dimensionality is reduced to 1 (i.e. slicing), then that dimension coord gets demoted to an auxiliary (scalar) coord."

This may be another special-case decision that Iris makes : That you don't generally want size-1 cube dimensions.
So we remove them when slicing ?sometimes? (but you can have such a thing, if you want).
However, this behaviour is, at least. consistent with what CF says about scalar coordinates.

"The use of scalar coordinate variables is a convenience feature which avoids adding size one dimensions to variables."

However, I think when slicing (as opposed to extract, maybe??), the behaviour (whether a dimension is reduced or 'lost') is actually consistent with numpy:

>>> print(cube)
unknown / (unknown)                 (y: 3; x: 4)
    Dimension coordinates:
        y                             x     -
        x                             -     x
>>> 
>>> print(cube[0:1])
unknown / (unknown)                 (y: 1; x: 4)
    Dimension coordinates:
        y                             x     -
        x                             -     x
>>> 
>>> print(cube[0])
unknown / (unknown)                 (x: 4)
    Dimension coordinates:
        x                             x
    Scalar coordinates:
        y                           0
>>> 

[pp-mo]

Hi @kaedonkers do we still think there is something here to address, and an improvement to be got out of this ?
Clearly, confusion occurs here and we could do something to reduce it.

My reading is still that Iris behaviour is defensible, but the details + especially terminology is confusing 😕 and sometimes surprising 😮 💣

So, I am still wondering whether we can add a docs section that will help.
I think it's far too much to hope to prevent confusion -- not least because, if you know something of CF, you probably thought you understood it already. But we could at least have something to point people at, when inevitable confusion occurs ...
So, we do already have a special section on how Iris translates to/from PP+FF files (^) ... and I think this info could belong in a similar overview account of how Iris represents CF data.
Do you think that could that work for this ?

Actually, it would be good to write a more generic account of file format interfaces, that would also cover GRIB, but for now I guess we should avoid "the perfect being the enemy of the good" ™️

(^) BTW in the latest (revamped) docs, the 'technical papers' have now been moved into the Developer Guide, which I think should maybe be changed, since they are technical info for the user rather than for devs.

[pp-mo]

@SciTools/peloton looks like this has been idle for a while.
Let's close now -- please do reopen @kaedonkers if you still feel strongly about this.