Skip to content

ARROW-1585/ARROW-1586: [PYTHON] serialize_pandas roundtrip loses columns name #1161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

ARROW-1585/ARROW-1586: [PYTHON] serialize_pandas roundtrip loses columns name #1161

wants to merge 1 commit into from

Conversation

@cpcloud
Copy link
Contributor

@cpcloud cpcloud commented Oct 4, 2017 •
edited
Loading

This PR also adds support for MultiIndex columns.

@wesm
Copy link
Member

wesm commented Oct 4, 2017

Can you rebase this on upstream?

@cpcloud
Copy link
Contributor Author

cpcloud commented Oct 4, 2017

Yep, on it.

wesm
wesm reviewed Oct 4, 2017
View reviewed changes
Copy link
Member

@wesm wesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How hard would it be to handle ARROW-1585 (integer columns) in this?

python/pyarrow/pandas_compat.py Outdated
Copy link
Member

@wesm wesm Oct 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this dtype need to be stored in the same way that we're storing normal pandas column dtypes (with logical + physical)? So maybe this will be a dict

@cpcloud
Copy link
Contributor Author

cpcloud commented Oct 4, 2017

I think I can handle ARROW-1585 in this PR.

@wesm
Copy link
Member

wesm commented Oct 4, 2017

Odd failure. the manylinux1 build is using pandas 0.20.1 in case it's hitting a pandas bug

@cpcloud cpcloud changed the title ARROW-1586: [PYTHON] serialize_pandas roundtrip loses columns name ARROW-1585/ARROW-1586: [PYTHON] serialize_pandas roundtrip loses columns name Oct 5, 2017
wesm
wesm approved these changes Oct 6, 2017
View reviewed changes
Copy link
Member

@wesm wesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, modulo minor comment

python/pyarrow/pandas_compat.py Outdated
Copy link
Member

@wesm wesm Oct 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this do the right thing for extension dtypes (e.g. categorical)?

@cpcloud
Copy link
Contributor Author

cpcloud commented Oct 6, 2017

@wesm This now handles categorical and datetimetz dtypes in column indexes.

@wesm
Copy link
Member

wesm commented Oct 6, 2017

Sweet thanks!!!

@wesm
Copy link
Member

wesm commented Oct 6, 2017

This is failing on py2 =|

@cpcloud
Copy link
Contributor Author

cpcloud commented Oct 7, 2017

@wesm This is now passing on Travis. Waiting for appveyor to start: https://ci.appveyor.com/project/cpcloud/arrow/build/1.0.329

@wesm
Copy link
Member

wesm commented Oct 7, 2017

Sweet, will keep an eye on the build

@asfgit asfgit closed this in e31c2e3 Oct 7, 2017
@wesm
Copy link
Member

wesm commented Oct 7, 2017

Thanks!

@cpcloud cpcloud deleted the ARROW-1586 branch October 7, 2017 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wesm wesm wesm approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cpcloud @wesm