BUG: pivot_table with nested elements and numpy 1.24

[mroeschke]

• closes BUG: numpy 1.24.0 causes ValueError with pivot_table and ragged nested sequences #50342 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

I think we have still pinned numpy in our 1.5.x branch but confirmed this still works locally

% conda list numpy
# packages in environment at /opt/miniconda3/envs/pandas-dev:
#
# Name                    Version                   Build  Channel
numpy                     1.24.1           py38hc2f29e8_0    conda-forge
numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge

% pytest pandas/tests/reshape/test_pivot.py -k test_pivot_table_with_mixed_nested_tuples
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: , configfile: pyproject.toml
plugins: asyncio-0.20.2, cython-0.2.0, hypothesis-6.58.1, xdist-3.0.2, cov-4.0.0, anyio-3.6.2
asyncio: mode=strict
collected 539 items / 538 deselected / 1 selected

pandas/tests/reshape/test_pivot.py .

------------------------------------------------------- generated xml file: /test-data.xml --------------------------------------------------------
================================================================================== slowest 30 durations ==================================================================================
0.01s call     pandas/tests/reshape/test_pivot.py::TestPivotTable::test_pivot_table_with_mixed_nested_tuples

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================================================================== 1 passed, 538 deselected in 0.48s ============================================================================

EDIT: Numpy is no longer pinned on main so this should run in the CI

[mroeschke]

Decided not to check the numpy version per se since in prior numpy versions this was raising a DeprecationWarning internally

[phofl]

Any idea about performance impact here?

[mroeschke]

Yeah in the worst case where this can't short circuit there's a perf hit

In [1]: values = list(range(1000))

In [2]: %timeit pd.core.common.asarray_tuplesafe(values) # PR
279 µs ± 1.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [2]: %timeit pd.core.common.asarray_tuplesafe(values) # main
41.8 µs ± 623 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

[phofl]

Is this a common use-case?

Would try-except be significantly faster?

[mroeschke]

Good idea. With the try except I get closer to main performance (~44)

[datapythonista]

lgtm, added two minor suggestions

[datapythonista]

In general I think it's better to just wrap the line that can raise in the try block. Is it a problem with the warning catching in this case? No big deal, I guess the warnings stuff won't raise a ValueError, but if it does, the behavior won't be as expected.

[phofl]

Hm in this case I think this is better as is, if we would wrap try-except into the catch_warnings statement, then we would also catch warnings in the except block, which isn't what we want here

[mroeschke]

Yeah result = np.asarray(values, dtype=dtype) will raise a warning (due to our usage) w/ numpy < 1.24 and raise an exception with numpy >= 1.24. As mentioned, I don't want to accidentally mask a warning within the except block.