BUG/ENH: consistent gzip compression arguments

[twoertwein]

• closes Deterministic gzip compressed outputs #28103
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

to_csv let's the user set all keyword arguments for gzip. Depending on whether the user provides a filename or a file object different keyword arguments can be set (gzip.open vs gzip.GzipFile).

This PR always uses gzip.GzipFile. The additional keyword arguments valid for gzip.open but not valid for gzip.GzipFile (encoding, errors, and newline) are still accessible:

pandas/pandas/io/common.py

Line 512 in aefae55

g = TextIOWrapper(f, encoding=encoding, errors=errors, newline="")

Using gzip.GzipFile, also allows us to set mtime to create reproducible gzip archives.

[arw2019]

lgtm

might we need to update the docstring or do you think it's good as is?

[twoertwein]

updating the doc string is a good idea, will do that! I assume that this will affect multiple to_* methods. Is there a good strategy instead of copy pasting the same docstring multiple times?

[arw2019]

updating the doc string is a good idea, will do that! I assume that this will affect multiple to_* methods. Is there a good strategy instead of copy pasting the same docstring multiple times?

You could maybe add the more explicit explanation to doc/source/user_guide/io.rst, under Quoting, compression, and file format, and add brief Changed in version 1.2.0 notes in docstrings of affected methods

[twoertwein]

The PR adding arguments for bz2/gzip #33398 mentioned that it affects to_csv, to_pickle, and to_json in the whatsnew but only updated the docstring for to_csv.

I could make sure that all three to_* methods have the same compression docstring and extend the mtime test case to also cover to_json and to_pickle.

[jreback]

looks good. sligthly OT, we want to add typing for the compression arg (I think we have an issue for this), similar to StorageOptions whereby we define it in pandas._typing.py

[jreback]

cc @gfyoung @WillAyd @TomAugspurger if comments.

[twoertwein]

looks good. sligthly OT, we want to add typing for the compression arg (I think we have an issue for this), similar to StorageOptions whereby we define it in pandas._typing.py

I will look into that, I assume it is going to be:

class CompressionArgs(TypedDict, total=False):
    method: str
    compresslevel: Optional[int]
    mtime:Optional[int]
    compression:int
    allowZip64:bool
    strict_timestamps:bool

technically, there are a few more but users should not pass them (filename, fileobj, buffer (deprecated since python 3.0), mode).

I could make sure that all three to_* methods have the same compression docstring and extend the mtime test case to also cover to_json and to_pickle.

Do you have opinions about that? compression does not only affect to_csv.

[WillAyd]

lgtm - nice PR

[pep8speaks]

Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-08-13 02:56:33 UTC

[twoertwein]

oh, I didn't know that TypedDict requires python 3.8. I will simply use Mapping[str, Optional[Union[str, int, bool]]].

[twoertwein]

probably unrelated to the recent CI issues, but we should definitely close those handles.

[jreback]

hmm, is there a ResoucceWarning?

[twoertwein]

I haven't seen any when reading/writing json files

[jreback]

thanks @twoertwein very nice!