REGR: Fix reading old pickles from pandas 1.3.5 #56308
Conversation
pandas/core/internals/blocks.py
Outdated
|
|
||
| # Old pandas (<=1.3.0) will call this function with placements | ||
| # that aren't necessarily BlockPlacements when unpickling | ||
| if not isinstance(placement, BlockPlacement): |
There was a problem hiding this comment.
can this be done in _setstate__ or something? a lot of effort has gone into optimizing this function
There was a problem hiding this comment.
also maybe deprecate reading of such-old pickles?
There was a problem hiding this comment.
Good idea on deprecating reading old pickles, I'll do that as a follow-up (1.3.5 is 2 years old at this point I think).
How would I change __setstate__? I'm not familiar at all with pickle stuff.
I thought __reduce__ calls new_block directly in 1.3.5.
pandas/pandas/_libs/internals.pyx
Lines 485 to 493 in 66e3805
There was a problem hiding this comment.
yah, __reduce__ sounds like the right place for this
There was a problem hiding this comment.
Hm, the old __reduce__ (from 1.3.5) is getting called though, which is leading to the problem, since it directly calls new_block.
The newer reduce goes through _unpickle_block, which does the conversion to BlockPlacement for us.
Are we OK just taking the perf hit for now?
There was a problem hiding this comment.
Might need patching in compat.pickle_compat then. That is always a PITA.
i think id be against taking the perf hit for something so old since it would be hit a zillion times.
|
Huh, the pickle_compat route is even simpler. I just re-routed |
|
Thanks so much for looking into this @lithomas1 |
…m pandas 1.3.5) (#56335) Backport PR #56308: REGR: Fix reading old pickles from pandas 1.3.5 Co-authored-by: Thomas Li <[email protected]>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.