Skip to content

PERF: avoid copies in frame[col].fillna(val, inplace=True) GH#46149 #46204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Mar 5, 2022
Merged

PERF: avoid copies in frame[col].fillna(val, inplace=True) GH#46149 #46204

merged 6 commits into from
Mar 5, 2022

Conversation

@jbrockmendel
Copy link
Member

@jbrockmendel jbrockmendel commented Mar 2, 2022

@pkuca
Copy link

pkuca commented Mar 3, 2022

Is this fix specific to fillna() usage with inplace=True? Would it also work with default inplace=False?

@jbrockmendel
Copy link
Member Author

jbrockmendel commented Mar 3, 2022

Would it also work with default inplace=False?

No.

@pkuca
Copy link

pkuca commented Mar 3, 2022

If that's out of scope for this PR I can open a new issue describing what I'm seeing after updating to 1.4.1.

It's actually the same as #46149, except my usage is:

df = df["field"].fillna("")

Of course I can just loc on pd.isna() and manually set to "" but it'd be great to not need to update all of our usages of fillna().

@jbrockmendel
Copy link
Member Author

jbrockmendel commented Mar 3, 2022

If that's out of scope for this PR I can open a new issue describing what I'm seeing after updating to 1.4.1.

Please do.

@jreback jreback added the Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate label Mar 3, 2022
@jreback
Copy link
Contributor

jreback commented Mar 3, 2022

@jbrockmendel ok to backport this if you can add a note in 1.4.2

@jreback jreback added this to the 1.4.2 milestone Mar 5, 2022
@jreback jreback merged commit 221c3aa into pandas-dev:main Mar 5, 2022
@jreback
Copy link
Contributor

jreback commented Mar 5, 2022

@meeseeksdev backport 1.4.x

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Mar 5, 2022
@jbrockmendel @meeseeksmachine
Backport PR pandas-dev#46204: PERF: avoid copies in frame[col].fillna…
df4e274 
…(val, inplace=True) GH#46149
@meeseeksmachine meeseeksmachine mentioned this pull request Mar 5, 2022
Merged
@lumberbot-app
Copy link

lumberbot-app bot commented Mar 5, 2022

Something went wrong ... Please have a look at my logs.

@jbrockmendel jbrockmendel deleted the perf-inplace branch March 5, 2022 23:54
jreback pushed a commit that referenced this pull request Mar 6, 2022
@meeseeksmachine @jbrockmendel
Backport PR #46204: PERF: avoid copies in frame[col].fillna(val, inpl…
1bcb2f9 
…ace=True) GH#46149 (#46242)

Co-authored-by: jbrockmendel <[email protected]>
@simonjayhawkins simonjayhawkins added the Regression Functionality that used to work in a prior pandas version label Mar 8, 2022
yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request Jul 13, 2022
@jbrockmendel @yehoshuadimarsky
PERF: avoid copies in frame[col].fillna(val, inplace=True) GH#46149 (p…
97f2446 
…andas-dev#46204)
@TendouArisu TendouArisu mentioned this pull request Mar 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Regression Functionality that used to work in a prior pandas version

Projects

None yet

Milestone

1.4.2

Development

Successfully merging this pull request may close these issues.

PERF: Series.fillna (that is part of DataFrame) with inplace=True high memory usage / memory leaking

4 participants

@jbrockmendel @pkuca @jreback @simonjayhawkins