Skip to content

Conversation

@sethmlarson
Copy link
Collaborator

Closes #91 This moves the pip SBOM discovery machinery from the CPython repository to this repository to not require pip maintainers to update the SBOM every time, saving difficulties with backporting and a bunch of manual effort.

There will be a follow-up PR to the CPython repository removing the machinery there once this PR lands.

The SBOM diff between running this script on Python-3.12.2.tgz:

4c4
<     "created": "2024-02-06T20:56:29Z",
---
>     "created": "2024-02-09T17:13:23Z",
7c7
<       "Tool: ReleaseTools-f39e1557464bc7d14019a88cb8257545ed4104f3\n"
---
>       "Tool: ReleaseTools-a5b55c248715c96d4d5207717bc2942a10b4b99d\n"
85980,85984d85979
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-cachecontrol",
<       "relationshipType": "DEPENDS_ON",
85990,85994d85984
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-certifi",
<       "relationshipType": "DEPENDS_ON",
86000,86004d85989
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-chardet",
<       "relationshipType": "DEPENDS_ON",
86010,86014d85994
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-colorama",
<       "relationshipType": "DEPENDS_ON",
86025,86029d86004
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-distlib",
<       "relationshipType": "DEPENDS_ON",
86035,86039d86009
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-distro",
<       "relationshipType": "DEPENDS_ON",
86055,86059d86024
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-idna",
<       "relationshipType": "DEPENDS_ON",
86080,86084d86044
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-msgpack",
<       "relationshipType": "DEPENDS_ON",
86090,86094d86049
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-packaging",
<       "relationshipType": "DEPENDS_ON",
86105,86109d86059
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-platformdirs",
<       "relationshipType": "DEPENDS_ON",
86115,86119d86064
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-pygments",
<       "relationshipType": "DEPENDS_ON",
86125,86129d86069
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-pyparsing",
<       "relationshipType": "DEPENDS_ON",
86135,86139d86074
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-pyproject-hooks",
<       "relationshipType": "DEPENDS_ON",
86145,86149d86079
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-requests",
<       "relationshipType": "DEPENDS_ON",
86155,86159d86084
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-resolvelib",
<       "relationshipType": "DEPENDS_ON",
86165,86169d86089
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-rich",
<       "relationshipType": "DEPENDS_ON",
86175,86179d86094
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-setuptools",
<       "relationshipType": "DEPENDS_ON",
86185,86189d86099
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-six",
<       "relationshipType": "DEPENDS_ON",
86195,86199d86104
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-tenacity",
<       "relationshipType": "DEPENDS_ON",
86205,86209d86109
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-tomli",
<       "relationshipType": "DEPENDS_ON",
86215,86219d86114
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-truststore",
<       "relationshipType": "DEPENDS_ON",
86225,86229d86119
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-typing-extensions",
<       "relationshipType": "DEPENDS_ON",
86235,86239d86124
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-urllib3",
<       "relationshipType": "DEPENDS_ON",
86241,86245d86125
<     },
<     {
<       "relatedSpdxElement": "SPDXRef-PACKAGE-webencodings",
<       "relationshipType": "DEPENDS_ON",
<       "spdxElementId": "SPDXRef-PACKAGE-cpython"

Notice this removes all the direct relationships between CPython and pip's subpackages, this is a good thing IMO since CPython doesn't directly depend on these packages. This still lets tools like scanners discover vulnerabilities because CPython still has a dependency relationship with pip.

raise ValueError(f"Couldn't fetch metadata for project '{project}' from PyPI: {e}")


def remove_pip_from_sbom(sbom_data: dict[str, typing.Any]) -> None:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be able to remove this function once I remove pip from the SBOM in the CPython source code.

"Tarball doesn't contain an SBOM at 'Misc/sbom.spdx.json'"
) from None
sbom_bytes = tarball.extractfile(sbom_tarball_member).read()
sbom_data = json.loads(sbom_bytes)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this variable name to match everywhere in this script.

Co-authored-by: Ezio Melotti <[email protected]>
Copy link
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be a follow-up PR to the CPython repository removing the machinery there once this PR lands.

And after that update https://devguide.python.org/developer-workflow/sbom/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move pip SBOM discovery to release-tools repository

3 participants