PI-2472: Tweak area weighting regrid move averaging out of loop #3596

abooton · 2019-12-12T09:15:28Z

Optimise the averaging in the _regrid_area_weighted_array.

This is achieved by refactoring the weights calculation, collating the data and the weights in a single array, then finally computing the weighted mean as a whole-array-manipulation.

Calculating the weights first is necessary because the size (i.e. number of elements) of the weights and source data region varies per target grid point. In order to record all the values in a single array, the maximum size of the weights array needs to be established, so the single array can be setup. Therefore we first computing the weights and record the maximum weight-array size, hence we have refactored the weights calculation too.

…nsions

ehogan · 2019-12-12T10:28:39Z

lib/iris/experimental/regrid.py

+        src_data = np.expand_dims(src_data, axis=src_data.ndim)
+        x_dim = src_data.ndim - 1
+    # Move y_dim and x_dim to last dimensions
+    src_data = np.moveaxis(src_data, x_dim, -1)


Is it worth performing a check to determine whether an axis needs to be moved before moving it?

ehogan · 2019-12-12T10:29:17Z

lib/iris/experimental/regrid.py

+    # Move y_dim and x_dim to last dimensions
+    src_data = np.moveaxis(src_data, x_dim, -1)
+    if x_dim < y_dim:
+        src_data = np.moveaxis(src_data, y_dim - 1, -2)


Would it be possible to add a comment explaining that y gets shifted along one position as x moves to the end.

ehogan · 2019-12-12T11:18:51Z

lib/iris/experimental/regrid.py

+        new_data = np.moveaxis(new_data, -1, y_dim_orig)
+    else:
+        new_data = np.moveaxis(new_data, -2, y_dim_orig)
+        new_data = np.moveaxis(new_data, -1, x_dim_orig)


Consider using x_dim and y_dim rather than -1 and -2 here?

These last two loops can be consolidated, since new_data will end in [..., y, x] in either case.

The last two loops can't be consolidated; add a comment explaining why :)

ehogan · 2019-12-12T12:02:48Z

lib/iris/experimental/regrid.py

+        area_func,
+        circular=False,
+    ):
+        """ """


Missing docstring?

Add a simple doc string.

ehogan · 2019-12-12T12:25:52Z

lib/iris/experimental/regrid.py


    # Flag to indicate whether the original data was a masked array.
-    src_masked = ma.isMaskedArray(src_data)
+    src_masked = src_data.mask.any() if ma.isMaskedArray(src_data) else False


Comment: there's one test that fulfils this condition.

However, there isn't a test that uses a masked array but that doesn't have any masked points.

I've added an extra test.

Looks good, thanks!

ehogan · 2019-12-12T14:31:50Z

lib/iris/experimental/regrid.py

-                )
+                data = src_data[..., y_indices, x_indices]
+                Nx = data.shape[-1]
+                Ny = data.shape[-2]


len_x and len_y rather than Nx and Ny?

ehogan · 2019-12-12T15:17:15Z

lib/iris/experimental/regrid.py

+
+    # Ensure we have x_dim and y_dim.
+    x_dim_orig = copy.copy(x_dim)
+    y_dim_orig = copy.copy(y_dim)


Stephen queried the use of copy.copy and I agree it isn't required here.

ehogan · 2019-12-12T15:32:57Z

lib/iris/experimental/regrid.py

    # dtype when necessary.
    dtype = np.promote_types(src_data.dtype, np.float16)
+    # Create empty arrays to hold src_data per target point, and weights
+    src_area_datas = np.zeros(src_areas_shape, dtype=np.float64)


Should dtype above be used rather than np.float64?

I think as it standard the code is using the numpy default value of np.float64, so I've used this here. I don't think we should revisit the dtype handling in a follow up PR as I think it would be beneficial to improve it.

…ide loop

lib/iris/experimental/regrid.py

… extra comments

* PI-2472: Tweak area weighting regrid move xdim and ydim axes (#3594) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * PI-2472: Tweak area weighting regrid enforce xdim ydim (#3595) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * _regrid_area_weighted_array: Ensure x_dim and y_dim * PI-2472: Tweak area weighting regrid move averaging out of loop (#3596) * _regrid_area_weighted_array: Refactor weights and move averaging outside loop

* PI-2472: Tweak area weighting regrid move xdim and ydim axes (SciTools#3594) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * PI-2472: Tweak area weighting regrid enforce xdim ydim (SciTools#3595) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * _regrid_area_weighted_array: Ensure x_dim and y_dim * PI-2472: Tweak area weighting regrid move averaging out of loop (SciTools#3596) * _regrid_area_weighted_array: Refactor weights and move averaging outside loop

* Bump version to 2.4.0rc0. * unpin mpl (#3468) * Merge pull request #3301 from bayliffe/fastpercentilemethod_mask_test Analysis percentile method - update applicability test for fast_percentile_method * Have Travis test with iris-grib, remove problem tests (#3469) * Have Travis test with iris-grib, remove problem tests * mock GribInternalError correctly * Update license headers * account for changes in handling of grib message defaults * Test against the latest version of python-eccodes * Moved irir-grib skip to iris.tests * Merge pull request #2608 from cpelley/PICKLEABLE_FORMATS TEST: Extends #2569 to unpickle * _regrid_area_weighted_array: Move axes creation over which weights are calculated to before loop (#3519) * Purge iris.experimental.regrid np<1.7 support (#3539) * Add NameConstraint with relaxed name loading (#3463) * _regrid_area_weighted_array: move indices variable nearer to use (#3564) * _regrid_area_weighted_array: Tweak variable order to near other use in code (#3571) * Fix problems with export and echo command. (#3577) * Pushdocs fix2 (#3580) * Revert to single-line command for doctr invocation. * Added script comment, partly to force Github respin. * Bracketed six.moves and __future__ imports. * Fixes required due to the release of iris-grib v0.15.0 (#3582) * Fix python-eccodes pin in travis (#3593) * PI-2472: Optimise the area weighted regridding routine (#3598) * PI-2472: Tweak area weighting regrid move xdim and ydim axes (#3594) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * PI-2472: Tweak area weighting regrid enforce xdim ydim (#3595) * _regrid_area_weighted_array: Set axis order to y_dim, x_dim last dimensions * _regrid_area_weighted_array: Extra tests for axes ordering * _regrid_area_weighted_array: Ensure x_dim and y_dim * PI-2472: Tweak area weighting regrid move averaging out of loop (#3596) * _regrid_area_weighted_array: Refactor weights and move averaging outside loop * Pin pillow to make graphics tests work again. (#3630) * PI-2472: Area weighted regridding (#3623) * The Area-weights routine is refactored into the "__prepare" and "__perform" structure, in-line with our other regridding methods. * The area-weights are now computed in the "__prepare", so are calculated once. * The information required for checking the regridding weights and target grid info are now cached on the "regridder" object. * This is inline with the general use already described in the documentation. * pep8 conformance. * pep8 compliance. * Allow some 'key=None' args in Geostationary creation. (#3628) LGTM * Allow some 'key=None' args in Geostationary creation. * Integration test loading netcdf geostationary without offset properties. * pep8 conformance. * pep8 conformance. * pep8 conformance. * test_NameConstraint get mock from iris.tests. * Remove use of super() in _constraints.py for Py2 compatibility. * Updated license headers. * Updated iris-grib reference in extensions.txt. * Py2 support for iris-grib in Travis. * Updates for auto docs for Iris 2.4 release. * What's new entry to unpinning mpl. * Edited Py2 support for iris-grib in Travis. * Renamed whatsnew contributions folder for v2.4. * Hacked tests.integration.test_grib2 to avoid import error from iris-grib version < 0.15. * Only test grib with python 3. * Compiled v2.4 whatsnew. Co-authored-by: Martin Yeo <[email protected]> Co-authored-by: Bill Little <[email protected]> Co-authored-by: stephenworsley <[email protected]> Co-authored-by: abooton <[email protected]> Co-authored-by: lbdreyer <[email protected]> Co-authored-by: Emma Hogan <[email protected]>

abooton added 3 commits December 8, 2019 09:59

_regrid_area_weighted_array: Set axis order to y_dim, x_dim last dime…

264d33b

…nsions

_regrid_area_weighted_array: Extra tests for axes ordering

0792ad1

_regrid_area_weighted_array: Ensure x_dim and y_dim

7d99492

This was referenced Dec 12, 2019

PI-2472: _regrid_area_weighted_array: Move averaging out of loop #3589

Closed

Cache for area-weighted regridder #2472

Closed

abooton added this to the v3.0.0 milestone Dec 12, 2019

ehogan reviewed Dec 12, 2019

View reviewed changes

ehogan mentioned this pull request Dec 12, 2019

PI-2472: Tweak area weighting regrid move xdim and ydim axes #3594

Merged

ehogan reviewed Dec 12, 2019

View reviewed changes

_regrid_area_weighted_array: Rebasing

b64d6f7

ehogan suggested changes Dec 12, 2019

View reviewed changes

_regrid_area_weighted_array: Refactor weights and move averaging outs…

852c34d

…ide loop

abooton force-pushed the tweak_area_weighting_regrid_move_averaging_out_of_loop branch from 4ffcdb7 to b3e0a5c Compare December 12, 2019 21:43

stickler-ci reviewed Dec 12, 2019

View reviewed changes

lib/iris/experimental/regrid.py Outdated Show resolved Hide resolved

_regrid_area_weighted_array: Only move x and y dims if necessary. Add…

2d13339

… extra comments

abooton force-pushed the tweak_area_weighting_regrid_move_averaging_out_of_loop branch from b3e0a5c to 2d13339 Compare December 12, 2019 21:54

_regrid_area_weighted_array: Resolving conflicts from merge

cd31d53

abooton merged commit c17a048 into SciTools:area_weighted_regridding Dec 13, 2019

abooton deleted the tweak_area_weighting_regrid_move_averaging_out_of_loop branch December 20, 2019 20:42

PI-2472: Tweak area weighting regrid move averaging out of loop #3596

PI-2472: Tweak area weighting regrid move averaging out of loop #3596

Uh oh!

Conversation

abooton commented Dec 12, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehogan Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ehogan Dec 12, 2019 •

edited

Loading