API for fetching latest and stable versions

[techtonik]

What's the problem this feature will solve?

Tools could use simple lookup logic for checking their latest and stable versions. The way it is currently done in pip is crazy:

https://github.com/pypa/pip/blob/35d51f1f405f6f31d68c800e73307686ec029254/src/pip/_internal/utils/outdated.py#L104-L118

Describe the solution you'd like

It should be as simple as fetching https://pypi.org/pypi/pip/latest/json or https://pypi.org/pypi/pip/stable/json endpoints.

[bskinn]

I don't know what fraction of PyPI's bandwidth is consumed by JSON requests, but I feel a bit bad pulling down the entire /pypi/{pkg}/json endpoint for a project/package when all I want is to check the most recent version number.

[di]

@bskinn Most of the bandwidth is consumed by distributions (files), not JSON. And even then, 90-something percent of it is cached by our CDN -- so don't feel bad 🙂

That said, this API makes sense to me and would be relatively easy to implement. The hard part is getting it right.

Something like the following seems to cover all use cases to me:

• /latest/json Latest non-prerelease if available, otherwise the latest prerelease
• /latest-stable/json Latest non-prerelease
• /latest-unstable/json Latest version including prereleases

Should just be redirects to the actual versions? E.g. /pypi/pip/latest/json -> /pypi/pip/20.1.1/json?

[bskinn]

<nod>, those three, and their names, make sense to me.

Main catch scenarios are (1) for all three, if no versions are available, and (2) specifically for latest-stable, in the case that no non-prerelease versions are available?

[di]

I imagine that they'd be 404 instead in that case, same as if the project didn't actually exist.

Another question, should any of these include yanked releases? I think not.

[bskinn]

Yeah, I figured they'd 404. Not knowing the warehouse codebase that well, though, I wasn't sure if there's already machinery there that would cleanly handle this particular failure mode & kick out the 404.

Agreed, I figure it should mimic the behavior of pip in dealing with yanked projects. Since this API call is intrinsically an operation where the user can't indicate a specific release, yanked releases should never be returned.

[bskinn]

If I read the model code correctly, Project.latest_version would supply the correct version for /latest/json/, including masking-out of any yanked releases.

If so (and if this is the right place in the codebase to implement the search), could just implement two variations that provide latest-stable and latest-unstable?

[di]

That would make sense to me!

[bskinn]

Ok... I've been poking at this, and I think I'll try to put together a PR for it.

One question, @di -- what is the semantic meaning of a NULL value for Release.is_prerelease?

Project.latest_version uses .nullslast(), which seems to suggest that there may be some edge cases where returning such a Release is desired. But, I'm wondering whether a new Project.latest_stable_version should also use .nullslast() in the same way, or if it should exclude .is_prerelease NULLs from the query altogether.

Regardless of the above, I figure Project.latest_unstable_version should follow .latest_version and also use .nullslast().

[di]

As far as I can tell, it should never actually return NULL. It's a column property:

https://github.com/pypa/warehouse/blob/bd2b3a22f4f84072eb03abb771a88ddab1489e7a/warehouse/packaging/models.py#L357

defined by this function:

https://github.com/pypa/warehouse/blob/1fbb4ac752e68b5840b9e09b68e44a165569bfa6/warehouse/migrations/versions/e7b09b5c089d_add_pep440_is_prerelease.py#L29-L34

but Release.version can't be NULL:

https://github.com/pypa/warehouse/blob/bd2b3a22f4f84072eb03abb771a88ddab1489e7a/warehouse/packaging/models.py#L355

[di]

Looks like it was added in #3470

[bskinn]

Maybe it's a guard against a mangled table? Regardless, looks like there's no need to consider the NULL case; and shouldn't hurt anything to just keep the .nullslast() in the call flow, in case. Thanks!