Skip to content

Converting pandas index takes very long, add in arrow_table. #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 24, 2023
Merged

Converting pandas index takes very long, add in arrow_table. #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d4ec637
Converting pandas index takes very long, add in arrow_table.
justinGilmer Jul 24, 2023
7f7b5f9
Bump pyproject toml version number for pip build.
justinGilmer Jul 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions btrdb/transformers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,10 +224,7 @@ def arrow_to_dataframe(
tmp = tmp_table.select(["time", *usable_cols])
else:
tmp = tmp_table
df = tmp.to_pandas(date_as_object=False, types_mapper=pd.ArrowDtype)
df = df.set_index("time")
df.index = pd.DatetimeIndex(df.index, tz="UTC")
return df
return tmp.to_pandas(date_as_object=False, types_mapper=pd.ArrowDtype)


def to_dataframe(streamset, columns=None, agg="mean", name_callable=None):
Expand Down Expand Up @@ -668,5 +665,7 @@ class StreamSetTransformer(object):
to_polars = to_polars
arrow_to_polars = arrow_to_polars

arrow_to_arrow_table = arrow_to_arrow_table

to_csv = to_csv
to_table = to_table
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "btrdb"
version = "5.30.1"
version = "5.30.2"
authors = [
{name="PingThingsIO", email="[email protected]"},
]
Expand Down