Automate NEWS/CHANGES Generation and Ensure Entries

[dstufft]

Currently pip has a CHANGES.txt file which the author of a change needs to add any changes too. However it's really easy to forget to mention changes there. In addition the fact that multiple authors are all editing that one file means that merge conflicts regularly happen in that file.

It would be great to automate the creation of this file as much as possible and do it in a way where we can test that a PR adds an entry to it.

I've seen three methods for automating this file:

• Derive it from the commit log, each commit represents an entry in the change log.
• Derive it from Github PR, each PR represents an entry in the change log.
• Use a separate file for each entry and combine them into one file as part of the release process.

Between these three methods, deriving it from the commit log means that we get an "automatic" change log, however I believe that in order to effectively use this method we'll need to use something other than Github. This is because we'll want something like Gerrit where we're shown the commit message in a way that makes it easier to review that and we'll want something that does squash merges or single commit per change instead of how Github does it with PRs. I also worry that it becomes near impossible to fix typos or reword a commit message in this style.

The second method is attractive for a similar reason as the first one is, we'll get automate change log entries from the PRs. However a lot of PRs simply do not have great titles or bodies which is kind of a bummer. On the plus side we can edit PR titles/bodies so we can change them at anytime, but on the negative side pretty much only we and the original author can do that and there's no history behind it. I also worry that this ties our release process directly to the Github API.

The third method is my personal favorite, it does require that contributors write explicit change log entries, however we can pretty easily test to ensure that this was done. It makes it easy for anyone to change a commit message after the fact and it comes with history. It also does not tie our release process directly to the Github API the same way as the second method does, nor does it require us to move away from Github PRs like the first method does.

[xavfernandez]

To check the changelog I've come up with a small script https://github.com/xavfernandez/pip/blob/changelog_checker/tasks/check_changelog.py that for each commit since the last pushed tag:

• check for issue/PR in the form of #XXXX in its changelog
• finds its related PR via the Github API
• extract the changelog from the CHANGES.txt file
• regroup commits in 3 categories:
  • commits with PR and whose related issue/PR number are found in the changelog (nothing to do for the reviewer)
  • commits without PR, there are usually few of those mainly version bump, etc (so usually quick to scan)
  • finally commits with PR but which are not present in the changelog (those need review)
• and print all of those with the extra informations

It helps to quickly scan the commit messages of all the commits without changelog.

To run the script, you just need github3.py and a token with public_repo access.
https://dpaste.de/RSLF shows an example output of python tasks/check_changelog.py 9.0.0

To complete this tool/ease the review, we could add a special GitHub label (changelog ok, etc) to the merged PR that does not require an additional entry in the changelog.

Feedback very welcome !

[RonnyPfannschmidt]

for evaluation - openstack made a tool called reno for composing user-facing changelogs

[xavfernandez]

It looks like reno is implementing Use a separate file for each entry and combine them into one file as part of the release process.

In the meantime, I've added the changelog:ok label, to mark merged PR as not needing changelog and updated https://github.com/xavfernandez/pip/blob/changelog_checker/tasks/check_changelog.py accordingly.

[dstufft]

This is done now.