Cell measure not used by collapsed

[znicholls]

Apologies if I've missed something in the documentation or the issues. I've read through everything I could find and still can't see how to address this and hence think it's an issue with the library (which is why this is here and not in the Google Group, happy to move if you'd like).

I'm trying to use information from a netCDF file with areacella information to take a weighted mean of some spatial data. When I do the weighted mean using iris.analysis.cartography.area_weights then it all works smoothly. However if I add a cell_measure to my cube, I get the warning

UserWarning: Collapsing spatial coordinate 'latitude' without weighting

All help is greatly appreciated. I put the minimal code to explain the error here.

from os.path import join
import iris

tas_file = join(
    "..",
    "tests",
    "test_data",
    "cmip5",
    "1pctCO2",
    "Amon",
    "tas",
    "CanESM2",
    "r1i1p1",
    "tas_Amon_CanESM2_1pctCO2_r1i1p1_185001-198912.nc",
)

tas_cube = iris.load_cube(tas_file)

areacealla_file = join(
    "..",
    "tests",
    "test_data",
    "cmip5",
    "1pctCO2",
    "fx",
    "areacella",
    "CanESM2",
    "r0i0p0",
    "areacella_fx_CanESM2_1pctCO2_r0i0p0.nc",
)

areacella_cube = iris.load_cube(areacealla_file)

areacella_measure = iris.coords.CellMeasure(
    areacella_cube.data, 
    standard_name=areacella_cube.standard_name, 
    long_name=areacella_cube.long_name, 
    var_name=areacella_cube.var_name, 
    units=areacella_cube.units, 
    attributes=areacella_cube.attributes, 
    measure='area'
)

lat_lon_dims = [
    i
    for i, a in enumerate(tas_cube.dim_coords)
    if a.standard_name in ['latitude', 'longitude']
]
tas_cube.add_cell_measure(
    areacella_measure,
    data_dims=lat_lon_dims
)
print(tas_cube)

air_temperature / (K)               (time: 1680; latitude: 64; longitude: 128)
     Dimension coordinates:
          time                           x               -              -
          latitude                       -               x              -
          longitude                      -               -              x
     Cell Measures:
          cell_area                      -               x              x
     Scalar coordinates:
          height: 2.0 m
     Attributes:
          CCCma_data_licence: 1) GRANT OF LICENCE - The Government of Canada (Environment Canada) is...
          CCCma_parent_runid: IGA
          CCCma_runid: IDK
          Conventions: CF-1.4
          associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CanESM2_1pctCO2_r0i0p0.nc...
          branch_time: 171915.0
          branch_time_YMDH: 2321:01:01:00
          cmor_version: 2.5.4
          contact: cccma_info@ec.gc.ca
          creation_date: 2011-03-10T12:09:13Z
          experiment: 1 percent per year CO2
          experiment_id: 1pctCO2
          forcing: GHG (GHG includes CO2 only)
          frequency: mon
          history: 2011-03-10T12:09:13Z altered by CMOR: Treated scalar dimension: 'height'....
          initialization_method: 1
          institute_id: CCCma
          institution: CCCma (Canadian Centre for Climate Modelling and Analysis, Victoria, BC,...
          model_id: CanESM2
          modeling_realm: atmos
          original_name: ST
          parent_experiment: pre-industrial control
          parent_experiment_id: piControl
          parent_experiment_rip: r1i1p1
          physics_version: 1
          product: output
          project_id: CMIP5
          realization: 1
          references: http://www.cccma.ec.gc.ca/models
          source: CanESM2 2010 atmosphere: CanAM4 (AGCM15i, T63L35) ocean: CanOM4 (OGCM4.0,...
          table_id: Table Amon (31 January 2011) 53b766a395ac41696af40aab76a49ae5
          title: CanESM2 model output prepared for CMIP5 1 percent per year CO2
          tracking_id: 36b6de63-cce5-4a7a-a3f4-69e5b4056fde
     Cell methods:
          mean: time (15 minutes)

# curiously, if I try to output (rather than print)
# this in a notebook I get an error, I will make 
# a separate issue later
tas_cube_mean = tas_cube.collapsed(
    ['latitude', 'longitude'],
    iris.analysis.MEAN,
)

/Users/zebedeenicholls/miniconda3/envs/scmcallib/lib/python3.6/site-packages/iris/cube.py:3174: UserWarning: Collapsing spatial coordinate 'latitude' without weighting
  warnings.warn(msg.format(coord.name()))

It is not clear to me how to use the area cell data and avoid this warning. Simply trying to pass in the cell measure as weights also fails as they need to be broadcast first I think.

tas_cube.collapsed(
    ['latitude', 'longitude'],
    iris.analysis.MEAN,
    weights=tas_cube.cell_measure("cell_area").data
)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-24-4996ed4d2a81> in <module>()
      2     ['latitude', 'longitude'],
      3     iris.analysis.MEAN,
----> 4     weights=tas_cube.cell_measure("cell_area").data
      5 )

~/miniconda3/envs/scmcallib/lib/python3.6/site-packages/iris/cube.py in collapsed(self, coords, aggregator, **kwargs)
   3271                 weights = kwargs["weights"].view()
   3272                 kwargs["weights"] = np.transpose(weights,
-> 3273                                                  dims).reshape(new_shape)
   3274 
   3275             data_result = aggregator.aggregate(unrolled_data,

~/miniconda3/envs/scmcallib/lib/python3.6/site-packages/numpy/core/fromnumeric.py in transpose(a, axes)
    573 
    574     """
--> 575     return _wrapfunc(a, 'transpose', axes)
    576 
    577 

~/miniconda3/envs/scmcallib/lib/python3.6/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     50 def _wrapfunc(obj, method, *args, **kwds):
     51     try:
---> 52         return getattr(obj, method)(*args, **kwds)
     53 
     54     # An AttributeError occurs if the object does not have

~/miniconda3/envs/scmcallib/lib/python3.6/site-packages/numpy/ma/core.py in wrapped_method(self, *args, **params)
   2573     """
   2574     def wrapped_method(self, *args, **params):
-> 2575         result = getattr(self._data, funcname)(*args, **params)
   2576         result = result.view(type(self))
   2577         result._update_from(self)

ValueError: axes don't match array

Double checking that the mean has actually been taken.

print(tas_cube_mean)

air_temperature / (K)               (time: 1680)
     Dimension coordinates:
          time                           x
     Scalar coordinates:
          height: 2.0 m
          latitude: 0.0 degrees, bound=(-90.0, 90.0) degrees
          longitude: 178.59375 degrees, bound=(-1.40625, 358.59375) degrees
    Scalar cell measures:
          cell_area
     Attributes:
          CCCma_data_licence: 1) GRANT OF LICENCE - The Government of Canada (Environment Canada) is...
          CCCma_parent_runid: IGA
          CCCma_runid: IDK
          Conventions: CF-1.4
          associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_CanESM2_1pctCO2_r0i0p0.nc...
          branch_time: 171915.0
          branch_time_YMDH: 2321:01:01:00
          cmor_version: 2.5.4
          contact: cccma_info@ec.gc.ca
          creation_date: 2011-03-10T12:09:13Z
          experiment: 1 percent per year CO2
          experiment_id: 1pctCO2
          forcing: GHG (GHG includes CO2 only)
          frequency: mon
          history: 2011-03-10T12:09:13Z altered by CMOR: Treated scalar dimension: 'height'....
          initialization_method: 1
          institute_id: CCCma
          institution: CCCma (Canadian Centre for Climate Modelling and Analysis, Victoria, BC,...
          model_id: CanESM2
          modeling_realm: atmos
          original_name: ST
          parent_experiment: pre-industrial control
          parent_experiment_id: piControl
          parent_experiment_rip: r1i1p1
          physics_version: 1
          product: output
          project_id: CMIP5
          realization: 1
          references: http://www.cccma.ec.gc.ca/models
          source: CanESM2 2010 atmosphere: CanAM4 (AGCM15i, T63L35) ocean: CanOM4 (OGCM4.0,...
          table_id: Table Amon (31 January 2011) 53b766a395ac41696af40aab76a49ae5
          title: CanESM2 model output prepared for CMIP5 1 percent per year CO2
          tracking_id: 36b6de63-cce5-4a7a-a3f4-69e5b4056fde
     Cell methods:
          mean: time (15 minutes)
          mean: latitude, longitude

Conda list output (in case it helps):

# packages in environment at /Users/zebedeenicholls/miniconda3/envs/scmcallib:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.3                      py_1    conda-forge
appnope                   0.1.0                    py36_0    conda-forge
asn1crypto                0.24.0                   py36_3    conda-forge
atomicwrites              1.2.1                    py36_0    conda-forge
attrs                     18.1.0                     py_1    conda-forge
automat                   0.7.0                      py_1    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    2.1.4                      py_1    conda-forge
bokeh                     0.13.0                   py36_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
cartopy                   0.16.0           py36h81b52dc_1    conda-forge
certifi                   2018.8.24                py36_1    conda-forge
cf_units                  2.0.1            py36h7eb728f_2    conda-forge
cffi                      1.11.5           py36h5e8e0c9_1    conda-forge
cftime                    1.0.1            py36h7eb728f_0    conda-forge
chardet                   3.0.4                    py36_3    conda-forge
clangdev                  6.0.1                 default_1    conda-forge
click                     6.7                        py_1    conda-forge
cloudpickle               0.5.5                      py_0    conda-forge
constantly                15.1.0                     py_0    conda-forge
coverage                  4.5.1                     <pip>
cryptography              2.3.1            py36hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                    py36_0    conda-forge
curl                      7.61.0               h93b3f91_1    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.9.0.1          py36h470a237_0    conda-forge
dask                      0.19.0                     py_0    conda-forge
dask-core                 0.19.0                     py_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
distributed               1.23.0                   py36_0    conda-forge
entrypoints               0.2.3                    py36_2    conda-forge
expat                     2.2.5                hfc679d8_1    conda-forge
ExpectException           0.1.1                     <pip>
f90nml                    1.0.2                     <pip>
freetype                  2.9.1                h6debe1e_1    conda-forge
geos                      3.6.2                hfc679d8_2    conda-forge
h5py                      2.8.0            py36hb794570_1    conda-forge
hdf4                      4.2.13                        0    conda-forge
hdf5                      1.10.2               hc401514_1    conda-forge
heapdict                  1.0.0                    py36_0    conda-forge
html5lib                  1.0.1                      py_0    conda-forge
hyperlink                 17.3.1                     py_0    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
idna                      2.7                      py36_2    conda-forge
incremental               17.5.0                     py_0    conda-forge
intel-openmp              2018.0.3                      0  
ipykernel                 4.8.2                    py36_0    conda-forge
ipython                   6.5.0                    py36_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.0                      py_0    conda-forge
iris                      2.1.0                    py36_3    conda-forge
jedi                      0.12.1                   py36_0    conda-forge
jinja2                    2.10                       py_1    conda-forge
joblib                    0.12.3                     py_0    conda-forge
jpeg                      9c                   h470a237_0    conda-forge
jsonschema                2.6.0                    py36_2    conda-forge
jupyter                   1.0.0                      py_1    conda-forge
jupyter_client            5.2.3                      py_1    conda-forge
jupyter_console           5.2.0                    py36_1    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
kiwisolver                1.0.1            py36h2d50403_2    conda-forge
krb5                      1.14.6                        0    conda-forge
libcxx                    6.0.1                         0    conda-forge
libffi                    3.2.1                hfc679d8_4    conda-forge
libgfortran               3.0.1                h93005f0_2  
libgpuarray               0.7.6                h470a237_3    conda-forge
libiconv                  1.15                 h470a237_2    conda-forge
libnetcdf                 4.6.1                h039f2a5_8    conda-forge
libopenblas               0.2.20               hdc02c5d_7  
libpng                    1.6.35               ha92aebf_0    conda-forge
libsodium                 1.0.16                        0    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libtiff                   4.0.9                he6b73bb_1    conda-forge
libxml2                   2.9.8                h422b904_3    conda-forge
libxslt                   1.1.32               h88dbc4e_1    conda-forge
llvm-meta                 6.0.1                         0    conda-forge
llvmdev                   6.0.1                hf8ce74a_2    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.2.4            py36hc9114bc_0    conda-forge
mako                      1.0.7                      py_1    conda-forge
markupsafe                1.0              py36h470a237_1    conda-forge
matplotlib                2.2.3            py36h0e0179f_0    conda-forge
mistune                   0.8.3            py36h470a237_2    conda-forge
mkl                       2018.0.3                      1  
mkl-service               1.1.2            py36h6b9c3cc_4  
more-itertools            4.3.0                    py36_0    conda-forge
msgpack-python            0.5.6            py36h2d50403_2    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
nbval                     0.9.1                     <pip>
nc_time_axis              1.1.0                      py_0    conda-forge/label/renamed
ncurses                   6.1                  hfc679d8_1    conda-forge
netcdf4                   1.4.1            py36h62672b6_0    conda-forge
netcdftime                1.0.0a2                  py36_0    conda-forge
notebook                  5.6.0                    py36_1    conda-forge
numpy                     1.14.3           py36he6379a5_1  
numpy-base                1.14.3           py36h7ef55bc_1  
olefile                   0.45.1                     py_1    conda-forge
openssl                   1.0.2o               h470a237_1    conda-forge
owslib                    0.16.0                     py_1    conda-forge
packaging                 17.1                       py_0    conda-forge
pandas                    0.23.4           py36hf8a1672_0    conda-forge
pandoc                    2.2.2                hde52d81_1    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.3.1                      py_0    conda-forge
partd                     0.3.8                      py_1    conda-forge
patsy                     0.5.0                      py_1    conda-forge
pexpect                   4.6.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pillow                    5.2.0            py36hc736899_1    conda-forge
pip                       18.0                     py36_1    conda-forge
pluggy                    0.7.1                      py_0    conda-forge
proj4                     4.9.3                h470a237_6    conda-forge
prometheus_client         0.3.0                      py_0    conda-forge
prompt_toolkit            1.0.15                     py_1    conda-forge
psutil                    5.4.7            py36h470a237_1    conda-forge
ptyprocess                0.6.0                    py36_0    conda-forge
py                        1.6.0                      py_0    conda-forge
pyasn1                    0.4.4                      py_0    conda-forge
pyasn1-modules            0.2.1                      py_0    conda-forge
pycparser                 2.18                       py_1    conda-forge
pyepsg                    0.3.2                      py_1    conda-forge
pygments                  2.2.0                      py_1    conda-forge
pygpu                     0.7.6            py36h7eb728f_0    conda-forge
pyhamcrest                1.9.0                      py_2    conda-forge
pyke                      1.1.1                    py36_0    conda-forge
pymagicc                  1.3.2+51.ga212ce1.dirty           <pip>
pymc3                     3.5                      py36_0    conda-forge
pyopenssl                 18.0.0                   py36_0    conda-forge
pyparsing                 2.2.0                      py_1    conda-forge
pyproj                    1.9.5.1                  py36_0    conda-forge
pyqt                      5.6.0            py36h8210e8a_7    conda-forge
pyshp                     1.2.12                     py_0    conda-forge
pysocks                   1.6.8                    py36_2    conda-forge
pytest                    3.7.4                    py36_0    conda-forge
python                    3.6.6                h5001a0f_0    conda-forge
python-dateutil           2.7.3                      py_0    conda-forge
pytz                      2018.5                     py_0    conda-forge
pyyaml                    3.13             py36h470a237_1    conda-forge
pyzmq                     17.1.2           py36hae99301_0    conda-forge
qt                        5.6.2                hd4c90f3_9    conda-forge
qtconsole                 4.4.1                    py36_1    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.19.1                   py36_1    conda-forge
scipy                     0.19.1           py36h3e758e1_3  
scmcallib                 0.0.1                     <pip>
send2trash                1.5.0                      py_0    conda-forge
service_identity          17.0.0                     py_0    conda-forge
setuptools                40.2.0                   py36_0    conda-forge
shapely                   1.6.4            py36h164cb2d_1    conda-forge
simplegeneric             0.8.1                      py_1    conda-forge
sip                       4.18.1           py36hfc679d8_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
sortedcontainers          2.0.4                      py_1    conda-forge
sqlite                    3.24.0               h2f33b56_0    conda-forge
tblib                     1.3.2                      py_1    conda-forge
terminado                 0.8.1                    py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
theano                    1.0.2                    py36_0    conda-forge
tk                        8.6.8                         0    conda-forge
toolz                     0.9.0                      py_0    conda-forge
tornado                   5.1              py36h470a237_1    conda-forge
tqdm                      4.24.0                     py_1    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
twisted                   18.7.0           py36h470a237_1    conda-forge
udunits2                  2.2.27.6             h3a4f0e9_0    conda-forge
urllib3                   1.23                     py36_1    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.31.1                   py36_1    conda-forge
widgetsnbextension        3.4.0                    py36_1    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
yaml                      0.1.7                h470a237_1    conda-forge
zeromq                    4.2.5                hfc679d8_5    conda-forge
zict                      0.1.3                      py_0    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
zope.interface            4.5.0            py36h470a237_1    conda-forge

[bjlittle]

@znicholls Hmmm this is pretty interesting...

I completely agree that your journey here is pretty painful... and to be honest it shouldn't be. That's completely the fault of iris, so rest assured you're not missing anything obvious - well, not that I know of 😉

Your use case highlights a couple of issues (for me):

• it should be easy to load data as a Cube and convert it to a CellMeasure, without having to manually jump through hoops i.e., say with a convenience method/function. As an aside, this should also be the case for Cube -> Coord, and vice versa, e.t.c.
• in general, iris should be able to automatically use the cell area data of a cell measure for spatial operations. This failing highlights a lack of joined up thinking in iris, particularly the lack of maturity for cell measure support internally.

@znicholls Would it be easy for you to somehow share your data for this use case, to help drive your issue forward?

[pp-mo]

@bjlittle it should be easy to load data as a Cube and convert it to a CellMeasure
... also ... for Cube -> Coord, and vice versa, e.t.c.

As you mention it, here are some ideas for that I developed a while back : #3485
Has the time come for this ??

[bjlittle]

@pp-mo Totally. I've been meaning to do something like this for years.

The interesting debate is whether we have a static factory method vs class factory method vs utility functions...

[znicholls]

Would it be easy for you to somehow share your data for this use case, to help drive your issue forward?

Yep sure. They all live in https://github.com/znicholls/netcdf-scm/tree/master/tests/test-data. The two files I've referenced above are https://github.com/znicholls/netcdf-scm/tree/master/tests/test-data/marble-cmip5/cmip5/1pctCO2/Amon/tas/CanESM2/r1i1p1 and https://github.com/znicholls/netcdf-scm/tree/master/tests/test-data/marble-cmip5/cmip5/1pctCO2/fx/areacella/CanESM2/r0i0p0 (sorry the names have changed slightly during other refactorings of that library).

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[github-actions]

This stale issue has been automatically closed due to a lack of community activity.

If you still care about this issue, then please either:

• Re-open this issue, if you have sufficient permissions, or
• Add a comment pinging @SciTools/iris-devs who will re-open on your behalf.