Loss of mask when slicing/extracting

[matthew-mizielinski]

When extracting a region, or slicing a dimension, from a cube with masked data the data type of the returned cube is not consistent with the source. Consider the following example;

import iris, numpy as np

print iris, iris.__version__
print np, np.__version__

masked_array = np.ma.MaskedArray([[1.,1.],[1.,100000.]], mask = [[False,False],[False,True]])
lat = iris.coords.DimCoord([-10,10],'latitude')
lon = iris.coords.DimCoord([0,180],'longitude')
cube = iris.cube.Cube(masked_array, dim_coords_and_dims = [(lat,0),(lon,1)])

cube_lat_slices = list(cube.slices_over('latitude'))

assert type(cube_lat_slices[0].data) == type(cube_lat_slices[1].data)

I would expect the cubes returned by slices_over to inherit the data type from cube, but when I run this code I get

<module 'iris' from ......./site-packages/iris/__init__.py'> 1.9.2
<module 'numpy' from '...../site-packages/numpy/__init__.pyc'> 1.8.2
Traceback (most recent call last):
  File "loss_of_masking.py", line 14, in <module>
    assert type(cube_lat_slices[0].data) == type(cube_lat_slices[1].data)
AssertionError

The first slice of cube has type numpy.ndarray, while the second has numpy.ma.core.MaskedArray.

The same issue arises when extracting a region from a masked cube that has no masked data (which is where the issue was first identified).

I view this behaviour as a (minor) bug and would suggest that the behaviour of iris should be made consistent with numpy in this situation; subsetting a MaskedArray returns a MaskedArray.

[ajdawson]

This has been discussed before in #1575. The conclusion was that this is intentional behaviour historically done this way for performance reasons, but there is suggestion that this may not be a concern any longer. I would prefer that masked arrays are persisted even if they have no masked values, which should be an easy change.

[matthew-mizielinski]

Is there any mileage in adding a configuration flag (similar to the ones under iris.FUTURE ) within iris to control which assumption is made?

[ajdawson]

Is there any mileage in adding a configuration flag (similar to the ones under iris.FUTURE ) within iris to control which assumption is made?

I don't think this is desirable, we should just aim to change the behaviour so we don't lose masks. In general we aim to avoid this type of configuration flag (the iris.FUTURE flags are for managing deprecations and will eventually disappear).

[cpelley]

I would prefer that masked arrays are persisted even if they have no masked values, which should be an easy change.

I'm assuming your proposition is to change the behaviour for realised data only?
I think being consistent with lazy data would be important in that case but this would not be so straight forward.

Biggus intentionally has no concept of whether its masked or not. You ask for what you want returned i.e. iris asks for a masked array, then determines if any values are masked to return a masked/unmasked array to the user. The issue is that end users might not always know whether their data is realised or not and so if we were to change the behaviour for realised data, then they will in all likelihood expect consistent behaviour with sliced lazy data too (i.e. if one slice is a masked array that all slices result in a masked array). This would require some fundamental changes to biggus.

[DPeterK]

Good news! This issue was fixed by #2856 🔍