Skip to content

Conversation

danchev
Copy link
Contributor

@danchev danchev commented Aug 16, 2025

Description

This PR fixes the broken build caused by the introduction of the --break-system-packages flag in pip.
The flag was introduced in 2023 as part of the implementation of PEP 668, which prevents pip from modifying externally managed Python environments by default.

Docker build currently fails because older versions of pip do not recognize this option.

Changes

Updated pip install commands to include --break-system-packages.

Added a note to ensure pip is upgraded to a recent version before invoking this flag.

Impact

Builds will succeed only when pip >= version supporting --break-system-packages is installed.

Environments running outdated pip will continue to fail until upgraded.

@github-actions github-actions bot added the devops improvements to build systems and github actions label Aug 16, 2025
@danchev danchev changed the title devops: Enable pip to modify EXTERNALLY-MANAGED Python installations Fix broken build: require updated pip to support --break-system-packages Aug 16, 2025
@danchev danchev marked this pull request as ready for review August 16, 2025 06:08
@Copilot Copilot AI review requested due to automatic review settings August 16, 2025 06:08
@danchev danchev requested a review from ngxson as a code owner August 16, 2025 06:08
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes the broken Docker build by ensuring compatibility with pip's --break-system-packages flag. The build was failing because older pip versions don't recognize this flag, which was introduced in 2023 as part of PEP 668 to prevent modifications to externally managed Python environments.

  • Upgraded pip to the latest version before installing dependencies
  • Added the --break-system-packages flag to bypass PEP 668 restrictions in containerized environments

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@@ -60,6 +60,7 @@ RUN apt-get update \
git \
python3 \
python3-pip \
&& pip install --upgrade pip setuptools wheel \
Copy link
Preview

Copilot AI Aug 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line continuation backslash is missing at the end of line 63, which will cause the Docker build to fail. Add a backslash () at the end of the line to properly continue the RUN command.

Copilot uses AI. Check for mistakes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be enough to upgrade only the pip package?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slaren, @danchev, @ngxson

Since the Github Actions for publishing the dockerimage specify the "ubuntu-22.04" base image, I think it's a good idea to remove the "--break-system-packages" option from ".devops/cuda.Dockerfile."

Alternatively, instead of removing the "--break-system-packages" option, I think the base image should be based on "ubuntu-24.04."

More details below.

The "--break-system-packages" option was added on "Aug 15, 2025, 12:45 AM GMT+9" via the following issue and pull request.

At this time, "Ubuntu The "--break-system-packages" option was added because container builds failed when using a base image with "ubuntu-24.04" specified, but the .github/workflows/docker.yml settings specifies a fixed value of "ubuntu-22.04", and the pip installed by python3-pip on ubuntu-22.04 is v22.x.

The "--break-system-packages" option is available in pip v23.1 and later, and installing pip v23.1 or later on "ubuntu-22.04" would likely be a bit of a hassle, and the impact would be significant if the base image were based on "ubuntu-24.04", so for the time being it is probably best to remove the "--break-system-packages" option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snmagn @slaren @ngxson

I apologize for the confusion. To be clear, the Docker build is currently failing not because of the addition of the --break-system-packages flag, but because the same PR/commit that introduced the flag also removed the line:
&& pip install --upgrade pip setuptools wheel \ from .devops/cuda.Dockerfile.

This PR contains two commits:

  1. A revert of the change that broke the build.
  2. A separate commit that correctly adds the --break-system-packages flag, which was the intended change in the original PR fix compile bug when the BASE_CUDA_DEV_CONTAINER is based on Ubuntu 2… #15005.

Effectively, this PR does not introduce anything new to cuda.Dockerfile; it simply restores a line that was mistakenly removed while still keeping the intended flag addition.

…stem installation)

- Updated pip install commands to include the --break-system-packages
  flag, ensuring compatibility when working with system-managed Python
  environments (PEP 668).

- Note: The --break-system-packages option was introduced in 2023.
  Ensure pip is updated to a recent version before using this flag.

fixes [ggml-org#15004](https://github.com/danchev/llama.cpp/issues/15004)
@slaren slaren linked an issue Aug 16, 2025 that may be closed by this pull request
@slaren slaren merged commit 618575c into ggml-org:master Aug 18, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops improvements to build systems and github actions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Misc. bug: ci: Publish CUDA Docker image failed
3 participants