to_numpy improvements for views

[henryiii]

Currently, to_numpy just dumps the same histogram view that .view() produces when you do to_numpy. This is probably not what you'd expect; you are asking for numpy-style output and numpy would not give you a boost-histogram view. It should give .view().value for accumulators. If you want a boost-histogram view, you can always do:

_, *edges = h.to_numpy()
view = h.view()

We could add an errors method to match Uproot 4, and mplhep could use this to plot errors for both libraries.

@HDembinski Do you agree or have further input? Using .to_numpy() will be common in Uproot 4, and it would be nice to be as similar as reasonable, so a natural "upgrade" will be .to_boost() ... do work ... .to_numpy().

[HDembinski]

I don't really understand. The point of .to_numpy was that it gives you the view consistently for the edges and data, if you specify flow=True.

There is no equivalent numpy structure for a boost-histogram with accumulators, so I think it is fine if it returns something that you have to call .value etc on.

[henryiii]

The problem is that to_numpy returns an object type that NumPy would never return. If you want to fill a weighted histogram, and then pass it to existing code that expect a NumPy array, you can't do that with .to_numpy; you instead have to do this:

H, *edges = h.to_numpy()
if hasattr(H, "value"):
    H = H.value
    # If NumPy ever adds a .value for ndarrays, your plotting library or whatever will break.

We really should be checking the dtype here for the value field, rather than relying on this not being an ndarray - but that's even harder to write.

This is likely to really hit users coming from Uproot, who just open a histogram and get a Weight storage. They won't be able to use h.to_numpy() directly on a boost-histogram object to feed an existing API that expects NumPy arrays. You can always get a full view with .view(flow=True) that has the extra data - it's no different than what to_numpy returns now. And we can't expect uproot's .to_numpy() to produce a boost-histogram view.

.to_numpy() is already a lossy conversion; you lose the axes types, the metadata, and so forth.

[henryiii]

For example, currently mplhep breaks when you give it a boost-histogram with a Weight storage because it assumed that to_numpy returns a simple NumPy array of values instead of a custom view and custom datatype. It works directly on an Uproot histogram, however, because that to_numpy returns a simple array and discards the errors.

Edit: or would break if it does a lossless conversion and errors are included. Not sure if they are yet.

[henryiii]

PS: keep in mind that accessing the "value" in the record array is a nocopy operation, so .to_numpy would still not make copies in memory.

[HDembinski]

The important aspect for me of to_numpy() is to get bin edges that match the flow setting. I am unwilling unhappy to give that up.

I don't remember and don't want to look up how the accumulator storage views are implemented, but if they are proper subclasses of ndarray, we should be able to make them return .value when client code does np.asarray on them. That would solve both problems, no?

[HDembinski]

It works directly on an Uproot histogram, however, because that to_numpy returns a simple array and discards the errors.

Which is pretty bad, because the errors are essential for any use.

.to_numpy() is already a lossy conversion; you lose the axes types, the metadata, and so forth.

There is a difference between losing metadata and actual data, like the variances.

[HDembinski]

"With proper subclass" I am referring to https://numpy.org/devdocs/user/basics.subclassing.html, which I studied recently. You can make a subclass that gracefully downgrades to a normal ndarray. In this case, the downgrade would imply returning .value. That would be a generally useful feature, I think. The views (in their downgraded form) would automatically work with all ufuncs then.

[henryiii]

View is already a proper subclass of ndarray. They are just a view into the exact data structure , with additions to make it easier to work with. We are slowly gaining full ufunct support, for example, so that views can be manipulated and retain correct variance calculations, etc. I don't think we should damage .view(), since that is doing just that - providing an exact view of the memory. However, I'm not completely against this, especially since I originally argued for something like this and you were strongly against it.

A few possibilities that are less intrusive:

• .to_numpy could gain an argument view=True, which would cause it to return the view instead of just the values (the default, to match other libraries like Uproot 4). Or it could be a selection, like ret="view", or ret="errors" - this could be synchronized with Uproot 4, which currently has a way to return the values and the errors instead of just the values if you pass errors=True. Consistency is good! If this returns a boost-histogram view by default, even a downgrading one, we will never be consistent with other libraries. We can add custom function parameters, though, like this one - we do the same thing with np.histogram(histogram=bh.Histogram).
• We could allow just the histogram to downgrade to values when calling np.asarray() on it. This cannot return the specialized subclass, so it's not as useful as view is. (doesn't solve the current problem, just mentioning it here)
• We could add a toggle so this view only downgrades, and regular views do not. But this still means that users have to call np.asarray(h) on the first result of to_numpy, which is counter intuitive and will cause things to break if they forget it. And it doesn't provide access to the errors in a uniform way (see my issue I am about to open).

I'll post one more idea for downstream libraries that might need to be taken into account when making a decision here. #423

Also, any results from these ideas/discussions are not intended to affect boost-histogram 0.10.

[HDembinski]

However, I'm not completely against this, especially since I originally argued for something like this and you were strongly against it.

It is not a great solution, that was probably because why I was against it. Silently dropping part of the data is not ideal. I think an additional keyword for .to_numpy() would be the best compromise, then?

[HDembinski]

To further elaborate on my position: I value consistency highly, so if our to_numpy() does something unexpected compared to the other scikit-hep libraries, then I am ok with changing our to_numpy(), but I'd still argue that there is no guideline what to do when a non-trivial storage is used and I question the choice of uproot. Returning something that discards the uncertainty estimates is fairly useless for any practical use case.

We should make a decision before version 1.0.