ENH: fill_value argument for shift #15486 #24128

ahcub · 2018-12-06T11:34:43Z

closes ENH: fill_value argument for shift #15486
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

!I master -> fork

pep8speaks · 2018-12-06T11:34:55Z

Hello @ahcub! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on December 26, 2018 at 00:17 Hours UTC

codecov · 2018-12-06T11:51:10Z

Codecov Report

Merging #24128 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24128      +/-   ##
==========================================
+ Coverage   92.29%    92.3%   +0.01%     
==========================================
  Files         162      163       +1     
  Lines       51832    51978     +146     
==========================================
+ Hits        47839    47980     +141     
- Misses       3993     3998       +5

Flag	Coverage Δ
#multiple	`90.71% <100%> (+0.01%)`	⬆️
#single	`42.99% <23.33%> (+0.01%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/series.py	`93.72% <ø> (+0.01%)`	⬆️
pandas/core/arrays/base.py	`97.59% <100%> (+0.17%)`	⬆️
pandas/core/frame.py	`96.91% <100%> (ø)`	⬆️
pandas/core/generic.py	`96.62% <100%> (-0.04%)`	⬇️
pandas/core/arrays/period.py	`98.48% <100%> (-0.02%)`	⬇️
pandas/core/internals/blocks.py	`93.81% <100%> (ø)`	⬆️
pandas/core/groupby/groupby.py	`96.8% <100%> (+0.15%)`	⬆️
pandas/core/arrays/categorical.py	`95.34% <100%> (+0.02%)`	⬆️
pandas/core/arrays/sparse.py	`92.1% <100%> (+0.01%)`	⬆️
pandas/core/arrays/datetimes.py	`97.6% <0%> (-0.64%)`	⬇️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6cf7d9...a03cbf5. Read the comment docs.

codecov · 2018-12-06T11:51:10Z

Codecov Report

Merging #24128 into master will decrease coverage by 49.17%.
The diff coverage is 58.33%.

@@             Coverage Diff             @@
##           master   #24128       +/-   ##
===========================================
- Coverage    92.2%   43.03%   -49.18%     
===========================================
  Files         162      162               
  Lines       51701    51701               
===========================================
- Hits        47672    22247    -25425     
- Misses       4029    29454    +25425

Flag	Coverage Δ
#multiple	`?`
#single	`43.03% <58.33%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/frame.py	`35.88% <100%> (-61.03%)`	⬇️
pandas/core/internals/blocks.py	`52.93% <50%> (-40.79%)`	⬇️
pandas/core/series.py	`49.32% <50%> (-44.38%)`	⬇️
pandas/core/generic.py	`39.65% <50%> (-57%)`	⬇️
pandas/core/arrays/datetimelike.py	`48.74% <66.66%> (-47.61%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/core/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
... and 124 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b841374...8b5cedb. Read the comment docs.

pandas/core/frame.py

pandas/core/generic.py

jreback · 2018-12-06T12:25:05Z

pandas/tests/frame/test_timeseries.py

@@ -320,6 +320,19 @@ def test_shift_categorical(self):
        xp = DataFrame({'one': s1.shift(1), 'two': s2.shift(1)})
        assert_frame_equal(rs, xp)

+    def test_shift_fill_value(self):
+        df = DataFrame(np.random.randnint(5), index=date_range('1/1/2000',


add the gh number as a comment

TomAugspurger

Probably need to add to Categorical, SparseArray. What about groupby?

pandas/core/internals/blocks.py

ahcub · 2018-12-07T00:47:09Z

added fill_value to shift implementations across different classes, according to the failures from tests

ahcub · 2018-12-08T00:03:02Z

@jreback I made all the changes, except the Index subclasses, because I think Index doesn't need fill_value. it does the shift correctly already with the freq specified
can you review?

mroeschke · 2018-12-25T19:28:09Z

One last linting error as well:

2018-12-25T18:31:36.1417763Z ERROR: /home/vsts/work/1/s/pandas/tests/arrays/sparse/test_array.py Imports are incorrectly sorted.
2018-12-25T18:31:36.1417924Z ERROR: /home/vsts/work/1/s/pandas/core/arrays/base.py Imports are incorrectly sorted.

ahcub · 2018-12-25T20:32:51Z

used isort to fix those, hopefully no issues this time 🙏

ahcub · 2018-12-25T21:19:22Z

just added the whatsnew note, not sure if it is ok
all the other checks were green

ahcub · 2018-12-25T22:19:52Z

all green @jreback

jreback

2 tiny corrections. ping on green and will merge. thanks for being so responsive.

jreback · 2018-12-25T23:30:20Z

doc/source/whatsnew/v0.24.0.rst

@@ -31,6 +31,7 @@ New features
 - :func:`read_feather` now accepts ``columns`` as an argument, allowing the user to specify which columns should be read. (:issue:`24025`)
 - :func:`DataFrame.to_html` now accepts ``render_links`` as an argument, allowing the user to generate HTML with links to any URLs that appear in the DataFrame.
  See the :ref:`section on writing HTML <io.html>` in the IO docs for example usage. (:issue:`2679`)
+- :meth:`DataFrame.shift` :meth:`Series.shift`, :meth:`ExtensionArray.shift`, :meth:`SparseArray.shift`, :meth:`Period.shift`, :meth:`GroupBy.shift`, :meth:`Categorical.shift`, :meth:`NDFrame.shift` and :meth:`Block.shift` now accept fill_value as an argument, allowing the user to specify a value which will be used instead of NA/NaT in the empty periods


can you use double backticks on fill_value and add a reference to the issue.

jreback · 2018-12-25T23:31:38Z

pandas/tests/arrays/sparse/test_array.py

@@ -9,6 +9,8 @@
 from pandas.compat import range
 import pandas.util._test_decorators as td

+from pandas.core.dtypes.missing import isna


in tests just
from pandas import isna

jreback · 2018-12-26T00:17:58Z

@ahcub lgtm. ping on green.

ahcub · 2018-12-26T00:53:23Z

all green @jreback

jreback · 2018-12-26T01:30:07Z

thanks @ahcub

this was a pretty involved change, thanks for sticking with it!

ahcub · 2018-12-26T01:38:22Z

thank you for all the support!

ahcub added 2 commits December 6, 2018 12:28

Merge pull request #3 from pandas-dev/master

c3e550c

!I master -> fork

ENH: fill_value argument for shift

9ab2d4d

ahcub added 5 commits December 6, 2018 13:37

!U remove the data frame part form the time series

6b4789d

!U remove the data frame part form the time series

57c087e

!U fix formatting

1852ea6

!U add test for data_frame

3461271

!U remove fill_value from tshift

8b5cedb

!U remove unrelevant change

27226fa

ahcub changed the title ~~ENH: fill_value argument for shift~~ ENH: fill_value argument for shift #15486 Dec 6, 2018

jreback requested changes Dec 6, 2018

View reviewed changes

jreback added Enhancement Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Dec 6, 2018

ahcub added 6 commits December 6, 2018 14:32

!U add doc for new fill_value argument for shift method

29b8a6e

!U add GH number

cf6b2dd

!U add versionated tag

9f9962c

!U add fill_value to the group by interface

4c7b762

!U replace np.nan to None

b36ca43

!U replace np.nan to None

a36e7c7

TomAugspurger reviewed Dec 6, 2018

View reviewed changes

pandas/core/internals/blocks.py Show resolved Hide resolved

!U add fill_value to the locations where it was missing

4c2ec2c

ahcub added 4 commits December 7, 2018 03:55

!U fix series test

fbf3d73

!U fix method name and values comparison for the new test

bb722a6

!U fix dataframe constructor

1b1fdc2

!U change the assert method used for new tests

c3f462c

ahcub added 5 commits December 25, 2018 20:10

parametrize on fill value with decorator instead of the test internals

bb67905

change check for is None to isna

f5aad59

change check for is None to isna

025f0db

change check for is None to isna

8ce460a

change check for is None to isna

dce5aa1

ahcub added 4 commits December 25, 2018 22:05

fix imports sorting

d60632a

fix imports sorting

a0ab35d

fix imports sorting

fd23842

fix imports sorting

3503e86

Add the whatsnew entry

674f15d

ahcub added 4 commits December 25, 2018 23:21

change func to meth

3c896ab

change the description in whatsnew

b9d335a

fix typo

4fcca01

Add categorical to the description

a24f5c8

jreback requested changes Dec 25, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Dec 25, 2018

ahcub added 2 commits December 26, 2018 02:14

change import of isna in test

a92ddfb

add backticks for fill_value and add the issue number

a03cbf5

jreback approved these changes Dec 26, 2018

View reviewed changes

jreback merged commit bf31c04 into pandas-dev:master Dec 26, 2018

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: fill_value argument for shift pandas-dev#15486 (pandas-dev#24128)

f3516fb

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

ENH: fill_value argument for shift pandas-dev#15486 (pandas-dev#24128)

e66bb18

Uh oh!

ENH: fill_value argument for shift #15486 #24128

ENH: fill_value argument for shift #15486 #24128

Uh oh!

Conversation

ahcub commented Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pep8speaks commented Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated on December 26, 2018 at 00:17 Hours UTC

Uh oh!

codecov bot commented Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codecov bot commented Dec 6, 2018

Codecov Report

Uh oh!

Uh oh!

Uh oh!

jreback Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

ahcub Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

TomAugspurger left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahcub commented Dec 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahcub commented Dec 8, 2018

Uh oh!

mroeschke commented Dec 25, 2018

Uh oh!

ahcub commented Dec 25, 2018

Uh oh!

ahcub commented Dec 25, 2018

Uh oh!

ahcub commented Dec 25, 2018

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

jreback Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

ahcub Dec 26, 2018

Choose a reason for hiding this comment

Uh oh!

jreback Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

ahcub Dec 26, 2018

Choose a reason for hiding this comment

Uh oh!

jreback commented Dec 26, 2018

Uh oh!

ahcub commented Dec 26, 2018

Uh oh!

jreback commented Dec 26, 2018

Uh oh!

ahcub commented Dec 26, 2018

Uh oh!

Uh oh!

ahcub commented Dec 6, 2018 •

edited

Loading

pep8speaks commented Dec 6, 2018 •

edited

Loading

codecov bot commented Dec 6, 2018 •

edited

Loading

TomAugspurger left a comment •

edited

Loading

ahcub commented Dec 7, 2018 •

edited

Loading