Reproducible installs

[ntninja]

What's the problem this feature will solve?

Reproducible installations. Currently when running pip install all relevant Python packages are installed into their respective and Python bytecode file are generated appropriately. Unfortunately these files are not reproducible (their contain the timestamps of the files they were generated for) and will therefor cause filesystem images they were created for to be non-reproducible as well.

Describe the solution you'd like
With Python 3.7 and PEP-552 a new and clean solution for this problem is now finally visible on the horizion.
Basically the call to py_compile.compile in PIP should be enhanced like this:

py_compile.compile(…, invalidation_mode=py_compile.PycInvalidationMode.UNCHECKED_HASH)

Since I'm guessing that PIP devs don't want to do this for all installations – there is no good reason IMHO, just assuming – another command-line flag will be required that allows one to the installation as reproducible as possible by enabling this flag.
(For full reproducibility installed shared libraries would require their timestamps to be zeroed as well, but I don't see how PIP can be any help in this currently.)

If the PIP team is willing to enable this by default, then it should only be enabled for non-editable system installs. Otherwise people will be surprised that their Python source changes are ignored by the interpreter.

My main (personal) use-case currently is docker-image-rebuilder: It runs a full docker build procedure then hashes the resulting filesystem and publishes the new version if there were any changes. PEP-552 also mentions build systems like Bazel and just about any Linux-distro as its use-cases. Most of these likely don't use PIP for gathering packages through.

Alternative Solutions
Since there are other non-reproducible files generated as well, I resort to filepath filtering rules for skipping this problem like "**/__pycache__/*.pyc" right now.

Additional context
Reproducible Builds (and, by extension, installs as well) are the future! 🙂

[mhsmith]

Alternatively, pip could just preserve the timestamps from extracted wheel files. The zipfile module can't do this itself, but it could be accomplished by adding a single line of code to unzip_file in https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py:

os.utime(fn, (time.time(), calendar.timegm(info.date_time)))

This will only help if you're installing from wheel files, but if you care about reproducibility then you should probably be doing that anyway.

[ntninja]

@mhsmith: True that! For wheel files this is probably the way to go. As far as I can tell the untar_file function already does in this line: tar.utime(member, path). So I guess what you're proposing should be a rather uncontroversial patch. 🙂

On the other hand, source installs are still important through: in my particular case many images are based on Alpine Linux, which uses musl-libc instead of glibc, so I cannot reuse -manylinux packages for binaries unfortunately. Also compiling from source IMHO makes sense here, since it's a compile-once/reuse-many kind of build, where reducing size is more important than build times.

[ntninja]

I created a PR regarding the low-level plumbing in distlib: https://bitbucket.org/pypa/distlib/pull-requests/38
Any comments/feedback on this would be apprechiated!

[jdemeyer]

Alternatively, pip could just preserve the timestamps from extracted wheel files.

Do you mean when installing a package from a wheel? How is that relevant to this issue? I'm not completely following.

[mhsmith]

The issue was that pip-generated .pyc files are not currently reproducible, because they contain embedded copies of the filesystem timestamps of the .py files they were built from.

[jdemeyer]

So you're saying that the actual bytes in a .pyc file depend on the timestamp of the corresponding .py file? That's strange...

[mhsmith]

It's so the interpreter can decide whether the file needs to be recompiled. Obviously this can give both false positives and false negatives, so PEP 552 specifes a replacement scheme which uses a content hash. This was implemented in Python 3.7, but is still not enabled by default.

[ntninja]

@jdemeyer: I think I should point out that @mhsmith and me are talking about two seperate but very closely related sub-issues with regards to reproducible installs:

1. I was talking about the .pyc files generated by PIP not being reproducible because they embed the modification timestamps of their respective .py files at all.
2. @mhsmith was talking about the .pyc files generated by PIP not being reproducible because they embed the modification timestamps of their respective .py files with those timestamps being different on each install.

My immediate use-case are sdist installs were the modification time of the installed file can indeed vary with each recompilation.
@mhsmith's immediate use-case are wheel installs were the modification times of the files in the wheel are currently not always retained upon installation (but have defined valued that could/should be retained).

Enabling installation with content-hashes (what this issue is mainly about) fixes the reproducibility issue in all cases, but is only available in Python 3.7+.
Properly extracting the mtimes from wheels (@mhsmith's original comment), only fixes the issue in that context; it will however work on all interpreter versions. (+ Not ignoring mtimes that are already present is the right thing to do in next to all cases anyways.)

[jdemeyer]

I wrote a proposal to distutils-sig for changing the timestamps of installed files. It's not directly related to this issue, but since it's about timestamps, so you may find it relevant.

[ntninja]

@jdemeyer: In general it feels like timestamps should be retained as much as possible (it should only change if somebody, or something, actually changed the file's contents).

Also the comparision with autotools is not really fair in the wheel case since autotools is a build-system, but wheels (by definition) are not built – they are only installed – and so no modification of files, and therefor timestamp changes, should take place. The only use-case I can find in your proposal for changing the timestamps regardless is to have a “when was this installed?” time of reference for recompilation tracking. Did you consider adding an extra …/installed marker file and touching/stating that as appropriate instead?

Regarding source installs I agree that preserving the timestamps is pretty problematic however (arbitrary files may be modified in unexpected ways). So restamping all files post-build seems like a reasonable idea; special-casing the .py files feels sub-optimal however. I not sure if there is any other solution however – you probably know that better than me.

^ Just my thoughts after reading your proposal. Maybe it's useful to you. 🙂

[jdemeyer]

Also the comparision with autotools is not really fair in the wheel case since autotools is a build-system, but wheels (by definition) are not built – they are only installed

My proposal concerns precisely the installation part, not the build part. So I don't see why wheels should be different from a from-source build.

[jdemeyer]

Did you consider adding an extra …/installed marker file and touching/stating that as appropriate instead?

No because that's completely incompatible with other build systems. The point here is to make Python-installed packages more similar to other-build-system installed packages.

[ntninja]

@jdemeyer: Because installation is fundamentally a non-mutating operation? („Take the files from there and put them here.“)
I also don't get why you say „I don't see why wheels should be different from a from-source build“ because there really is no reason. 🙂 When you take the source and build it into “binary” form (creating a wheel iirc) that obviously is a mutating operation, so retaining the previous timestamps doesn't really make sense (unless you really just copy-paste the files, maybe). But when you then install the generated wheel no files are actually changed anymore so why would you insistant on marking the files as changed, by chaning their timestamps, anyways?

Also IMHO a better comparision for the installation phase, in the PIP context, would be the Debian package manager: When you select a package to be installed it will simply assemble the list of packages, download them to the system, and then extract their contents to the filesystem – pretty much exactly what PIP does when it has wheels. (That fact that it will also run maintainer scripts is beside the point here.) The comparison with autotools doesn't even make sense unless you're talking about sdists. APT also retains the timestamps from packages it downloads.