[Python] pa.array([Decimal("nan")]) raises

[jbrockmendel]

Describe the bug, including details regarding any error messages, version, and platform.

pa.array([Decimal("nan")])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Decimal precision out of range [1, 38]: -2147483648

Based on how pyarrow handles numpy nans I'd expect this to get converted to a pyarrow null.

pyarrow 11.0.0 on mac

Component(s)

Python

[jorisvandenbossche]

Based on how pyarrow handles numpy nans I'd expect this to get converted to a pyarrow null.

We actually preserve NaNs by default when passing them like that:

In [16]: pa.array([float("nan")])
Out[16]: 
<pyarrow.lib.DoubleArray object at 0x7f41b7c90ee0>
[
  nan
]

It's only when you specify from_pandas=True that we convert NaN to null, but this argument is set to True automatically when you pass a pandas object (Series or Index, we should expand this with checking for an array as well ..), see https://arrow.apache.org/docs/python/generated/pyarrow.array.html

That aside, the error you get here is a bit confusing, and that seems to come from a bug in the precision/scale inference. If the first value is not a NaN, we see a better error message:

In [17]: pa.array([Decimal("1.20"), Decimal("nan")])
...
ArrowInvalid: The string 'NaN' is not a valid decimal128 number

So we generally don't support NaN (or +/- Inf) for decimal data. Given that we don't support it, we should maybe consider converting it to nulls instead (or at least give the option to do so). Also casting float to decimal will raise an error for NaN/Inf values:

In [32]: pa.array([1.2, 0.0]).cast(pa.decimal128(3, 2))
Out[32]: 
<pyarrow.lib.Decimal128Array object at 0x7f41b7fd8c40>
[
  1.20,
  0.00
]

In [33]: pa.array([1.2, 0.0, np.nan]).cast(pa.decimal128(3, 2))
...
ArrowInvalid: Cannot convert nan to Decimal128

For the array(..) constructor, the from_pandas=True argument will already ensure this gets converted to null:

In [34]: pa.array([Decimal("1.20"), Decimal("nan")], from_pandas=True)
Out[34]: 
<pyarrow.lib.Decimal128Array object at 0x7f41c415cac0>
[
  1.20,
  null
]

[jbrockmendel]

Thanks @jorisvandenbossche, using from_pandas should help with the motivating use case.

[jbrockmendel]

Hmm looks like pa.array([Decimal("nan")], from_pandas=True) gives a null-type instead of a decimal-type.

[jorisvandenbossche]

Yes, for a generic sequence of python objects, we currently just check if an object is "null" or not using:

arrow/python/pyarrow/src/arrow/python/helpers.cc

Lines 359 to 372 in 53c0d33

	bool PandasObjectIsNull(PyObject* obj) {
	if (!MayHaveNaN(obj)) {
	return false;
	}
	if (obj == Py_None) {
	return true;
	}
	if (PyFloat_IsNaN(obj) || (pandas_NA && obj == pandas_NA) ||
	(pandas_NaTType && PyObject_TypeCheck(obj, pandas_NaTType)) ||
	(internal::PyDecimal_Check(obj) && internal::PyDecimal_ISNAN(obj))) {
	return true;
	}
	return false;
	}

but once we determined an object to be null, we don't take the actual type of object into account for the type inference.

So that also means that currently you could mix all kinds of null-likes together when specifying from_pandas=True (also for cases where pandas would not allow this), for example:

In [18]: pa.array([1, pd.NaT], from_pandas=True)
Out[18]: 
<pyarrow.lib.Int64Array object at 0x7fcb37d62ec0>
[
  1,
  null
]

[jbrockmendel]

Makes sense, thanks. Context is is implementing something like pandas-dev/pandas#53025 for Decimal objects.

[xbarra]

In pyarrow 13 we have the same issue. And (the case most interesting to us), we have problems with Infinity

pa.array([Decimal(2),Decimal('inf'),3])

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[23], line 1
----> 1 pa.array([Decimal(2),Decimal('inf'),3])

File [c:\ProgramData\Anaconda3\Lib\site-packages\pyarrow\array.pxi:327](file:///C:/ProgramData/Anaconda3/Lib/site-packages/pyarrow/array.pxi:327), in pyarrow.lib.array()

File [c:\ProgramData\Anaconda3\Lib\site-packages\pyarrow\array.pxi:39](file:///C:/ProgramData/Anaconda3/Lib/site-packages/pyarrow/array.pxi:39), in pyarrow.lib._sequence_to_array()

File [c:\ProgramData\Anaconda3\Lib\site-packages\pyarrow\error.pxi:144](file:///C:/ProgramData/Anaconda3/Lib/site-packages/pyarrow/error.pxi:144), in pyarrow.lib.pyarrow_internal_check_status()

TypeError: 'str' object cannot be interpreted as an integer

The same happens with pa.array([Decimal(2),Decimal('inf'),3],from_pandas=True)

I suppose that in the end comes from decimal.cc ParseDecimalComponents and the DecimalComponents not having the options of storing nan and Infinity..