Dockerize CI for robust local reproducers

[powderluv]

Opening an issue to discuss the PR #1156 and capture it here so it can live beyond a PR.

[powderluv]

I would suggest we modify https://github.com/llvm/torch-mlir/blob/main/build_tools/python_deploy/build_linux_packages.sh
since it is used by the release scripts already. We should be able to expose the additional CI parameters

torch-mlir/.github/workflows/buildAndTest.yml

Lines 15 to 20 in 31727f8

	os: [macos-12, ubuntu-22.04]
	targetarch: [x86_64, AArch64]
	python-version: ["3.10"]
	torch-binary: [ON, OFF]
	llvmtype: [source, binary]
	llvmbuildtype: [in-tree, out-of-tree]

via any additional ENV variables.

Then when we get to run_on_docker() we branch out based on env vars to call the corresponding flavors of CI builds or Release builds. (

torch-mlir/.github/workflows/buildAndTest.yml

Lines 50 to 101 in 31727f8

	if: matrix.llvmtype == 'binary'
	run: |
	cd $GITHUB_WORKSPACE
	cmake $GITHUB_WORKSPACE/externals/llvm-project/llvm -B build -GNinja \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_LINKER=lld \
	-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
	-DPython3_EXECUTABLE=$(which python) \
	-DLLVM_ENABLE_ASSERTIONS=ON \
	-DLLVM_ENABLE_PROJECTS=mlir \
	-DLLVM_EXTERNAL_PROJECTS="torch-mlir;torch-mlir-dialects" \
	-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$GITHUB_WORKSPACE" \
	-DLLVM_EXTERNAL_TORCH_MLIR_DIALECTS_SOURCE_DIR="${GITHUB_WORKSPACE}/external/llvm-external-projects/torch-mlir-dialects" \
	-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
	-DTORCH_MLIR_ENABLE_MHLO=ON \
	-DTORCH_MLIR_USE_INSTALLED_PYTORCH=${{ matrix.torch-binary }} \
	-DCMAKE_OSX_ARCHITECTURES=${{ matrix.taregetarch }} \
	-DMACOSX_DEPLOYMENT_TARGET=10.15 \
	-DLLVM_TARGETS_TO_BUILD=host
	- name: llvm-source-out-of-tree-torch-src-or-binary
	# This build takes a while but is expected to almost always be cached.
	# A cache invalidation occurs when the committed LLVM version is changed.
	if: matrix.llvmtype == 'source'
	run: |
	cd $GITHUB_WORKSPACE
	cmake -Bllvm-build -GNinja \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_LINKER=lld \
	-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
	-DPython3_EXECUTABLE=$(which python) \
	-DLLVM_ENABLE_ASSERTIONS=ON \
	-DLLVM_ENABLE_PROJECTS=mlir \
	-DMLIR_ENABLE_BINDINGS_PYTHON=ON \
	-DLLVM_TARGETS_TO_BUILD=host \
	externals/llvm-project/llvm
	cmake --build llvm-build
	cmake -GNinja -Bbuild \
	-DCMAKE_LINKER=lld \
	-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
	-DMLIR_DIR="$(pwd)/llvm-build/lib/cmake/mlir/" \
	-DLLVM_DIR="$(pwd)/llvm-build/lib/cmake/llvm/" \
	-DMLIR_ENABLE_BINDINGS_PYTHON=OFF \
	-DTORCH_MLIR_ENABLE_MHLO=ON \
	-DTORCH_MLIR_USE_INSTALLED_PYTORCH=${{ matrix.torch-binary }} \
	-DTORCH_MLIR_ENABLE_LTC=OFF \
	-DPython3_EXECUTABLE=$(which python) \
	.

or the release build with

torch-mlir/build_tools/python_deploy/build_linux_packages.sh

Lines 101 to 105 in 31727f8

	python -m pip install -r /main_checkout/torch-mlir/requirements.txt --extra-index-url https://download.pytorch.org/whl/nightly/cpu
	CMAKE_GENERATOR=Ninja \
	TORCH_MLIR_PYTHON_PACKAGE_VERSION=${TORCH_MLIR_PYTHON_PACKAGE_VERSION} \
	python -m pip wheel -v -w /wheelhouse /main_checkout/torch-mlir/ \
	--extra-index-url https://download.pytorch.org/whl/nightly/cpu

)

Since the docker image itself is a variable you can override the environment variable with any CUDA docker image etc when you run it.

So locally when you want to build the CI versions you would do something like

TM_PYTHON_VERSIONS=cp310-cp310 MANYLINUX_DOCKER_IMAGE=nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04 TORCH_BINARY=OFF ./build_tools/python_deploy/build_linux_packages.sh

We can have a list for all targets etc so you don't have to run it one by one.

Couple points to note: We may have to expose the .ccache directory inside the docker container since our CIs use ccache extensively.

Let me know if this makes sense if not we can discuss at the community meeting too. I am trying to get the Release builds to move to PyTorch source builds but have to deal with nested pip sandboxes since we build our pip package and pytorch also with pip but that should be independent of these changes.

[sjain-stanford]

Thanks @powderluv . This proposal makes sense overall, and would be a lot more customizable with minimum boilerplate. The main concern I have though is it doesn't provide a good mechanism to repro the CI run locally (which was the intent of the original PR). Because it relies on the GitHub Actions step to install initial dependencies (that's how it avoids needing a Dockerfile). Do you think we could dockerize that part, so we have the reproducer for local runs? cc: @silvasean @asaadaldien .

[powderluv]

Yeah we can move the CI to run with the new docker images so they will all be reproducible locally.

[sjain-stanford]

Perfect. Just to confirm I understood:

1. We'd be using a Dockerfile to absorb these GHA setup steps:

  - name: Set up Python
    uses: actions/setup-python@v2
    with:
      python-version: 3.9
  - name: Install MLIR Python depends
    run: |
      python -m pip install -r $GITHUB_WORKSPACE/externals/llvm-project/mlir/python/requirements.txt
    shell: bash
  - name: Install PyTorch nightly depends
    run: |
      python -m pip install -r requirements.txt
    shell: bash
  - name: Install Ninja
    uses: llvm/actions/install-ninja@55d844821959226fab4911f96f37071c1d4c3268

2. This Dockerfile (potentially one per platform) is used to build and launch containers for building+testing torch-mlir (using shell scripts that run on GHA CI and locally alike).

Does that sound correct? I think this would be a big step forward in providing a consistent way to reproduce failures locally, and also make sending PRs upstream a lot easier. WDYT?

[powderluv]

Yes this sounds good. Just that the docker image we have already should be able to do all of this.
It has all versions of Python we care about, and installing requirements can be done for build configs that require it (running through setup.py automates some of those aspects).

So essentially all your changes can start with one file: torch-mlir/build_tools/python_deploy/build_linux_packages.sh

And you should get to a point where you can run all the equivalents of the CIs currently locally via that script. Once that script is good it is just a matter of updating GHA.

[sjain-stanford]

Makes sense. Thank you for the clarifications and guidance!

[powderluv]

We need to make sure we mount and hit ccache inside the docker when on GHA.

[sjain-stanford]

We need to make sure we mount and hit ccache inside the docker when on GHA.

After a few iterations on this, I am able to confirm that ccache is now configured to hit (~98.91% hit rate) even within the docker container on GHA. The CMake build step took <5 minutes (which otherwise needed >1.5 hours without ccache).

stats updated                       Sat Aug  6 14:27:28 2022
stats zeroed                        Sat Aug  6 14:20:32 2022
cache hit (direct)                  2178
cache hit (preprocessed)               3
cache miss                            24
cache hit rate                     98.91 %
unsupported code directive             1
cleanups performed                     0
files in cache                      6678
cache size                         498.6 MB
max cache size                       2.0 GB

In general I'm seeing good first-signs-of-life for the dockerized GHA CI run (all unit tests passing). Do you know why I still see some integration tests failing? Maybe something off with the python setup? I'll continue to cleanup (reduce/eliminate Dockerfile, move setup steps to shell scripts that are docker run etc) and then look into these failures more.

[powderluv]

are you on top of main that seems like an issue we just fixed