Implement split cube attributes. (#5040)

* Implement split cube attributes.

* Test fixes.

* Modify examples for simpler metadata printouts.

* Added tests, small behaviour fixes.

* Simplify copy.

* Fix doctests.

* Skip doctests with non-replicable outputs (from use of sets).

* Tidy test comments, and add extra test.

* Tiny typo.

* Remove redundant redefinition of Cube.attributes.

* Add CubeAttrsDict in module __all__ + improve docs coverage.

* Review changes - small test changes.

* More review changes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CubeAttrsDict example docstrings.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Odd small fixes.

* Improved docstrings and comments; fix doctests.

* Don't sidestep netcdf4 thread-safety.

* Publicise LimitedAttributeDict, so CubeAttrsDict can refer to it.

* Fix various internal + external links.

* Update lib/iris/cube.py

Co-authored-by: Martin Yeo <[email protected]>

* Update lib/iris/cube.py

Co-authored-by: Martin Yeo <[email protected]>

* Update lib/iris/cube.py

Co-authored-by: Martin Yeo <[email protected]>

* Update lib/iris/cube.py

Co-authored-by: Martin Yeo <[email protected]>

* Streamline docs.

* Review changes.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Martin Yeo <[email protected]>

Author: 3 people

docs/src/further_topics/metadata.rst

@@ -131,7 +131,7 @@ We can easily get all of the associated metadata of the :class:`~iris.cube.Cube`
 using the ``metadata`` property:
 
     >>> cube.metadata
-    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
+    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes=CubeAttrsDict(globals={}, locals={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}), cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
 
 We can also inspect the ``metadata`` of the ``longitude``
 :class:`~iris.coords.DimCoord` attached to the :class:`~iris.cube.Cube` in the same way:
@@ -675,8 +675,8 @@ For example, consider the following :class:`~iris.common.metadata.CubeMetadata`,
 
 .. doctest:: metadata-combine
 
-    >>> cube.metadata  # doctest: +SKIP
-    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
+    >>> cube.metadata
+    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes=CubeAttrsDict(globals={}, locals={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}), cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
 
 We can perform the **identity function** by comparing the metadata with itself,
 
@@ -701,7 +701,7 @@ which is replaced with a **different value**,
     >>> metadata != cube.metadata
     True
     >>> metadata.combine(cube.metadata)  # doctest: +SKIP
-    CubeMetadata(standard_name=None, long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'STASH': STASH(model=1, section=3, item=236), 'source': 'Data from Met Office Unified Model 6.05', 'Model scenario': 'A1B', 'Conventions': 'CF-1.5'}, cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
+    CubeMetadata(standard_name=None, long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'Model scenario': 'A1B', 'STASH': STASH(model=1, section=3, item=236), 'source': 'Data from Met Office Unified Model 6.05'}, cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
 
 The ``combine`` method combines metadata by performing a **strict** comparison
 between each of the associated metadata member values,
@@ -810,16 +810,17 @@ the ``from_metadata`` class method. For example, given the following
 
 .. doctest:: metadata-convert
 
-    >>> cube.metadata  # doctest: +SKIP
-    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
+    >>> cube.metadata
+    CubeMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes=CubeAttrsDict(globals={}, locals={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}), cell_methods=(CellMethod(method='mean', coord_names=('time',), intervals=('6 hour',), comments=()),))
 
 We can easily convert it to a :class:`~iris.common.metadata.DimCoordMetadata` instance
 using ``from_metadata``,
 
 .. doctest:: metadata-convert
 
-    >>> DimCoordMetadata.from_metadata(cube.metadata)  # doctest: +SKIP
-    DimCoordMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, coord_system=None, climatological=None, circular=None)
+    >>> newmeta = DimCoordMetadata.from_metadata(cube.metadata)
+    >>> print(newmeta)
+    DimCoordMetadata(standard_name=air_temperature, var_name=air_temperature, units=K, attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'})
 
 By examining :numref:`metadata members table`, we can see that the
 :class:`~iris.cube.Cube` and :class:`~iris.coords.DimCoord` container
@@ -849,9 +850,9 @@ class instance,
 
 .. doctest:: metadata-convert
 
-    >>> longitude.metadata.from_metadata(cube.metadata)
-    DimCoordMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, coord_system=None, climatological=None, circular=None)
-
+   >>> newmeta = longitude.metadata.from_metadata(cube.metadata)
+   >>> print(newmeta)
+   DimCoordMetadata(standard_name=air_temperature, var_name=air_temperature, units=K, attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'})
 
 .. _metadata assignment:
 
@@ -978,7 +979,7 @@ Indeed, it's also possible to assign to the ``metadata`` property with a
     >>> longitude.metadata
     DimCoordMetadata(standard_name='longitude', long_name=None, var_name='longitude', units=Unit('degrees'), attributes={}, coord_system=GeogCS(6371229.0), climatological=False, circular=False)
     >>> longitude.metadata = cube.metadata
-    >>> longitude.metadata  # doctest: +SKIP
+    >>> longitude.metadata
     DimCoordMetadata(standard_name='air_temperature', long_name=None, var_name='air_temperature', units=Unit('K'), attributes={'Conventions': 'CF-1.5', 'STASH': STASH(model=1, section=3, item=236), 'Model scenario': 'A1B', 'source': 'Data from Met Office Unified Model 6.05'}, coord_system=GeogCS(6371229.0), climatological=False, circular=False)
 
 Note that, only **common** metadata members will be assigned new associated

lib/iris/common/metadata.py

@@ -242,7 +242,11 @@ def __str__(self):
         field_strings = []
         for field in self._fields:
             value = getattr(self, field)
-            if value is None or isinstance(value, (str, dict)) and not value:
+            if (
+                value is None
+                or isinstance(value, (str, Mapping))
+                and not value
+            ):
                 continue
             field_strings.append(f"{field}={value}")
 

lib/iris/common/mixin.py

@@ -17,7 +17,7 @@
 
 from .metadata import BaseMetadata
 
-__all__ = ["CFVariableMixin"]
+__all__ = ["CFVariableMixin", "LimitedAttributeDict"]
 
 
 def _get_valid_standard_name(name):
@@ -53,7 +53,29 @@ def _get_valid_standard_name(name):
 
 
 class LimitedAttributeDict(dict):
-    _forbidden_keys = (
+    """
+    A specialised 'dict' subclass, which forbids (errors) certain attribute names.
+
+    Used for the attribute dictionaries of all Iris data objects (that is,
+    :class:`CFVariableMixin` and its subclasses).
+
+    The "excluded" attributes are those which either :mod:`netCDF4` or Iris intpret and
+    control with special meaning, which therefore should *not* be defined as custom
+    'user' attributes on Iris data objects such as cubes.
+
+    For example : "coordinates", "grid_mapping", "scale_factor".
+
+    The 'forbidden' attributes are those listed in
+    :data:`iris.common.mixin.LimitedAttributeDict.CF_ATTRS_FORBIDDEN` .
+
+    All the forbidden attributes are amongst those listed in
+    `Appendix A of the CF Conventions: <http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#attribute-appendix>`_
+    -- however, not *all* of them, since not all are interpreted by Iris.
+
+    """
+
+    #: Attributes with special CF meaning, forbidden in Iris attribute dictionaries.
+    CF_ATTRS_FORBIDDEN = (
         "standard_name",
         "long_name",
         "units",
@@ -78,7 +100,7 @@ def __init__(self, *args, **kwargs):
         dict.__init__(self, *args, **kwargs)
         # Check validity of keys
         for key in self.keys():
-            if key in self._forbidden_keys:
+            if key in self.CF_ATTRS_FORBIDDEN:
                 raise ValueError(f"{key!r} is not a permitted attribute")
 
     def __eq__(self, other):
@@ -99,11 +121,12 @@ def __ne__(self, other):
         return not self == other
 
     def __setitem__(self, key, value):
-        if key in self._forbidden_keys:
+        if key in self.CF_ATTRS_FORBIDDEN:
             raise ValueError(f"{key!r} is not a permitted attribute")
         dict.__setitem__(self, key, value)
 
     def update(self, other, **kwargs):
+        """Standard ``dict.update()`` operation."""
         # Gather incoming keys
         keys = []
         if hasattr(other, "keys"):
@@ -115,7 +138,7 @@ def update(self, other, **kwargs):
 
         # Check validity of keys
         for key in keys:
-            if key in self._forbidden_keys:
+            if key in self.CF_ATTRS_FORBIDDEN:
                 raise ValueError(f"{key!r} is not a permitted attribute")
 
         dict.update(self, other, **kwargs)