Timezones are not comparable between different pytz versions

[rockg]

I have HDF5 files created in 0.12 which were localized to a timezone ('US/Eastern'). When the dataframe is updated, it is done by calling combine_first which ultimately calls tseries.index.join. Now when I try to update in 0.14, the HDF5 representation changes. This is because of the different versions of pytz and timezone representation (see below), both sets of data will be converted to UTC and ultimately saved in UTC rather than US/Eastern. This certainly surprised me. This is using master branch as of this morning.

tz
Out[10]: <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD> #New data
data.index.tz
Out[11]: <DstTzInfo 'US/Eastern' EST-1 day, 19:00:00 STD> #Saved data
data.index.tz == tz
Out[12]: False

Also, in ipython typing "data.index.tz" gets stuck at "data.index." for about 20 seconds and my CPU spikes to 100% before completing. I don't know if this is simply related to the versions or something else.

[jreback]

when you are creating your current data you are not localizing (EST changed in pyt==2014.3 IIRC).

e.g. pytz.timezone('US/Eastern') is the current zone

[rockg]

I have two sets of data both localized to US/Eastern. One localized using pytz shipped with pandas 0.12 (saved to HDF5) and one localized using pytz shipped with pandas 0.14. Joining these loses all timezone information and converts to UTC.

[jreback]

pytz is not shipped with pandas

[rockg]

The pytz that was around when pandas 0.12 came out (you know what I mean).

If I understand correctly, you are saying that I have to go through and manually localize all the HDF5 files I have to be localized to the most current EST definition? Or I can probably just use the old pytz and avoid this hassle?

[jreback]

i would just use the same pytz.

[rockg]

K. I think so too. I wouldn't mind putting up a note somewhere in the documentation that notifies people of this potential issue (normally not an issue with data on the fly, but an issue when dealing with stored data).

[jreback]

sure that would be good.

though these should be equivalent (they won't compare == but they should represent the same dates).

[rockg]

Right, I guess that was my eventual point that they should be considered the same by pandas in some sense.

Any thoughts on the iPython issue? It's more like 50 seconds than 20.

[jreback]

no idea, do a %prun and post (and a reproducible exple if you can)