[BUG] Series.diff was always setting the dtype to object

[MarcoGorelli]

• closes Series lose nullable integer dtype after call to .diff() #30889
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[MarcoGorelli]

@jreback @jbrockmendel @WillAyd thanks for your quick reviews, I've updated the PR accordingly

[jorisvandenbossche]

Thanks for the PR!

[TomAugspurger]

Isn't the correct fix to to dispatch diff to the EA, rather than trying to fix the type after the fact? Or can we perhaps write a version of diff in terms of take and __sub__?

[jreback]

Isn't the correct fix to to dispatch diff to the EA, rather than trying to fix the type after the fact? Or can we perhaps write a version of diff in terms of take and __sub__?

this is actually a good idea

would be simple to write as a generic EA
method

[MarcoGorelli]

@TomAugspurger @jreback I don't understand.

Currently, we have

        result = algorithms.diff(com.values_from_object(self), periods)

In the case of

pd.Series([1, 2, 3], dtype="Int16").diff()

then

com.values_from_object(self)

is already converting the values to Object before even reaching algorithms.diff:

>>> com.values_from_object(self)
array([1, 2, 3], dtype=object)

So should com.values_from_object be changed?

Anyway, I realise that there's a somewhat tight deadline to get v1.0.0 out, and I'm happy to keep tackling this issue, but if you think it is (currently) beyond my capabilities, I'm happy for someone else to take over :)

[TomAugspurger]

So should com.values_from_object be changed?

We would need to not call that, at least for EAs, but probably for all dtypes, before passing it off. I can probably take a look tomorrow.

We have a week or two for 1.0.0 final.

[MarcoGorelli]

@TomAugspurger OK thanks - I might have figured out what you were suggesting anyway, have pushed another commit

[TomAugspurger]

See #31025 for an alternative which allows EAs to define how the operation should be done.

[TomAugspurger]

I think we'll proceed with #31025. Apologies for the duplicated effort @MarcoGorelli.

[MarcoGorelli]

It's OK, it was a good learning experience to attempt this and then read through your PR!