TST: errors in test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals)

[dhomeier]

Getting these errors in the 0.18.0rc1 test suite on Mac OS X 10.10 (numpy 0.11.0b3):

with Python 2.7.11:

ERROR: test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 290, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 276, in f
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 262, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5349, in _to_arrays
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5428, in _list_to_arrays
    coerce_float=coerce_float)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in _convert_object_array
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5492, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/common.py", line 1675, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 280, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 407, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/scratch.noindex/fink.build/pandas-py27-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 381, in _convert_listlike
    require_iso8601=require_iso8601
  File "pandas/tslib.pyx", line 1956, in pandas.tslib.array_to_datetime (pandas/tslib.c:39318)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2145, in pandas.tslib.array_to_datetime (pandas/tslib.c:38913)
    raise
  File "pandas/tslib.pyx", line 2139, in pandas.tslib.array_to_datetime (pandas/tslib.c:38803)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python2.7/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python2.7/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

with Python 3.4.4 and 3.5.1:

ERROR: test_constructor_compound_dtypes (pandas.tests.frame.test_block_internals.TestDataFrameBlockInternals)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2055, in pandas.tslib.array_to_datetime (pandas/tslib.c:37286)
    _string_to_dts(val, &dts, &out_local, &out_tzoffset)
  File "pandas/src/datetime.pxd", line 141, in datetime._string_to_dts (pandas/tslib.c:83021)
    raise ValueError('Unable to parse %s' % str(val))
ValueError: Unable to parse b'aa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2075, in pandas.tslib.array_to_datetime (pandas/tslib.c:37657)
    py_dt = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pandas/tslib.pyx", line 2081, in pandas.tslib.array_to_datetime (pandas/tslib.c:37756)
    raise TypeError("invalid string coercion to datetime")
TypeError: invalid string coercion to datetime

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 290, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tests/frame/test_block_internals.py", line 276, in f
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 262, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5349, in _to_arrays
    dtype=dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5428, in _list_to_arrays
    coerce_float=coerce_float)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in _convert_object_array
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5495, in <listcomp>
    arrays = [convert(arr) for arr in content]
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/frame.py", line 5492, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/core/common.py", line 1675, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 280, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 407, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/scratch.noindex/fink.build/pandas-py34-0.18.0rc1-1/pandas-0.18.0rc1/pandas/tseries/tools.py", line 381, in _convert_listlike
    require_iso8601=require_iso8601
  File "pandas/tslib.pyx", line 1956, in pandas.tslib.array_to_datetime (pandas/tslib.c:39318)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2145, in pandas.tslib.array_to_datetime (pandas/tslib.c:38913)
    raise
  File "pandas/tslib.pyx", line 2139, in pandas.tslib.array_to_datetime (pandas/tslib.c:38803)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1728, in pandas.tslib.parse_datetime_string (pandas/tslib.c:31663)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/sw/lib/python3.4/site-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

[jreback]

I cannot repro this (with the exact versions you are using)

I have PYTHONIOENCODING=UTF-8 set. (which is typical for a PY3 env)

actually this var doesn't matter

[jreback]

I have seen this happen on windows, IOW the dateparsing routines fail and raise a different error than we are expecting.

can you show versions of pytz, dateutil

(numpy_dev3)bash-3.2$ conda list pytz
# packages in environment at /Users/jreback/miniconda/envs/numpy_dev3:
#
pytz                      2015.7                   py35_0  
(numpy_dev3)bash-3.2$ conda list dateutil       
# packages in environment at /Users/jreback/miniconda/envs/numpy_dev3:
#
python-dateutil           2.4.2                    py35_0  

[dhomeier]

A bit behind, yes:

> fink list pytz dateutil
Information about 9337 packages read in 4 seconds.
 i   dateutil-py27                    2.2-3                    Extended date and time calculations
 i   dateutil-py34                    2.2-3                    Extended date and time calculations
 i   dateutil-py35                    2.2-3                    Extended date and time calculations
 i   pytz-py27                        2014.2-1                 World timezone definitions database
 i   pytz-py34                        2014.2-1                 World timezone definitions database
 i   pytz-py35                        2014.2-1                 World timezone definitions database

[jreback]

ok, that is it. dateutil==2.2 (via pip) reproduces.

[jreback]

Wish I had a better test, but this fixes. We are not testing this. As have to modify the ci to explicity pip install dateutil=2.2 (as its not on conda, as IIRC it was a completely buggy/bogus version :<). Its not even tagged in the repo, really weird

diff --git a/pandas/core/common.py b/pandas/core/common.py
index 70c02c5..0a6a309 100644
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -1681,7 +1681,7 @@ def _possibly_cast_to_datetime(value, dtype, errors='raise'):
                                 errors=errors).tz_localize(dtype.tz)
                         elif is_timedelta64:
                             value = to_timedelta(value, errors=errors)._values
-                    except (AttributeError, ValueError):
+                    except (AttributeError, ValueError, TypeError):
                         pass

         # coerce datetimelike to object

[dhomeier]

Tested the patch here as well, and also the original source against dateutil-2.4.2, both passing.

[jreback]

@dhomeier gr8 thanks!