[BUG] Cannot pick a single value if that value is overflow/underflow

[henryiii]

Currently, since using using Boost.Histogram's slicing as the backend, it has become impossible to "pick" an axis on an overflow or underflow bin. The following code fails:

def test_pick_flowbin():
    w = 1e-2  # e.g. a cross section for a process
    x = np.random.normal(loc=0.4, scale=0.4, size=1000)
    y = np.random.normal(loc=0.6, scale=0.4, size=1000)

    h = bh.Histogram(
        bh.axis.Regular(3, 0, 1),
        bh.axis.Regular(3, 0, 1),
        storage=bh.storage.Weight(),
    )
    h.fill(x, y, weight=w)

    uf_slice = h[bh.tag.underflow, ...]

The reason is that it is picking is implemented via a workaround - a single bin is sliced:

--------------------------------------------- Captured log call --------------------------------------------------------
DEBUG    boost_histogram._internal.hist:hist.py:778
                Reduce with [reduce_command(slice_and_rebin(iaxis=0, begin=-1, end=0, merge=1, mode=slice_mode.crop))]

For a flow bin, this is an invalid slice. This would be much easier with a "pick" of some sort from Boost.Histogram. Possibly the only fix for now would be to special case under/overflow, and implement some ugly logic for it.

Discovered on Gitter by @andrzejnovak.

[henryiii]

PS, for @HDembinski's benefit: we currently implement picking a single value as a 1 element cropping slice, then we do a project to remove the picked axes. So h[7, ...] translates roughly to h._reduce(slice_and_rebin(0,7,8,1,slice_mode.crop).project(*range(1,len(h))).

And multiple element picking on categorical axes, which is a slightly different concept, is not yet supported.

[andrzejnovak]

Btw with Variable axis this somehow works

import hist
import numpy as np
s = hist.tag.Slicer()

w = 1e-2  
x = np.random.normal(loc=0.4, scale=0.4, size=1000)
y = np.random.normal(loc=0.6, scale=0.4, size=1000)

h = hist.Hist(
    hist.axis.Variable([0, 0.5, 1], name="x", label="axis"),
    hist.axis.Regular(3, 0, 1, name="y", label="axis"),
    hist.storage.Weight()
    )
h.fill(x=x, y=y, weight=w)
h[hist.tag.underflow,  ...]

[henryiii]

You are correct:

@pytest.mark.parametrize(
    "ax",
    [bh.axis.Regular(3, 0, 1), bh.axis.Variable([0, 0.3, 0.6, 1])],
    ids=["regular", "variable"],
)
def test_pick_flowbin(ax):
    w = 1e-2  # e.g. a cross section for a process
    x = [-0.1, -0.1, 0.1, 0.1, 0.1]
    y = [-0.1, 0.1, -0.1, -0.1, 0.1]

    h = bh.Histogram(
        ax,
        ax,
        storage=bh.storage.Weight(),
    )
    h.fill(x, y, weight=w)

    uf_slice = h[bh.tag.underflow, ...]
    assert uf_slice.values(flow=True) == approx(np.array([1, 1, 0, 0, 0]) * w)

    uf_slice = h[..., bh.tag.underflow]
    assert uf_slice.values(flow=True) == approx(np.array([1, 2, 0, 0, 0]) * w)

>       reduced = self._hist.reduce(*slices)
E       ValueError: bins > 0 required

src/boost_histogram/_internal/hist.py:779: ValueError
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
DEBUG    boost_histogram._internal.hist:hist.py:778 Reduce with [reduce_command(slice_and_rebin(iaxis=0, begin=-1, end=0, merge=1, mode=slice_mode.crop))]
==================================================================== short test summary info ====================================================================
FAILED tests/test_histogram_indexing.py::test_pick_flowbin[regular] - ValueError: bins > 0 required
=========================================================== 1 failed, 780 passed, 1 skipped in 2.85s ============================================================

[andrzejnovak]

I am quite liking this "formulate issues as tests" 💯

[henryiii]

Adding pytest boilerplate to the issue template might be a good idea to push people in the right direction. :)

[HDembinski]

When writing unit tests, one needs to check all the corner cases, like this one.

I also stumbled over this bug, btw, I just did not have time to report it at the time.

[HDembinski]

reduce is not the right tool here. You can generate a new histogram with the axis removed and slice the view of the original histogram, then assign the outcome to the view of the new histogram.