Skip to content

Add support for publishing macOS M1 ARM64 wheels #2559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 10, 2021
Merged

Add support for publishing macOS M1 ARM64 wheels #2559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9634f7f
Add support for publishing macOS M1 ARM64 wheels
lgeiger Sep 7, 2021
a8e1695
Link against the tensorflow-macos when building for ARM Macs
lgeiger Sep 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 .github/workflows/make_wheel_Linux.sh → .github/workflows/make_wheel_Linux_x86.sh
View file
Open in desktop
File renamed without changes.
0 .github/workflows/make_wheel_Windows.sh → .github/workflows/make_wheel_Windows_x86.sh
View file
Open in desktop
File renamed without changes.
31 changes: 31 additions & 0 deletions .github/workflows/make_wheel_macOS_arm64.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
set -e -x

export TF_NEED_CUDA=0

python --version
python -m pip install --default-timeout=1000 delocate wheel setuptools tensorflow==$TF_VERSION

python configure.py

# For dynamic linking, we want the ARM version of TensorFlow.
# Since we cannot run it on x86 so we need to force pip to install it regardless
python -m pip install \
--platform=macosx_11_0_arm64 \
--no-deps \
--target=$(python -c 'import site; print(site.getsitepackages()[0])') \
--upgrade \
tensorflow-macos==$TF_VERSION

bazel build \
--cpu=darwin_arm64 \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CPU setting makes sure that we cross compile for ARM64.

--copt -mmacosx-version-min=11.0 \
--linkopt -mmacosx-version-min=11.0 \
Comment on lines +21 to +22
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to support 10.13 since the only the latest version macOS version currently supports ARM64. But setting a minimum version here makes sure that the script will keep working even if the GitHub actions VM are updated to 11.1 at some point in the future.

--noshow_progress \
--noshow_loading_progress \
--verbose_failures \
--test_output=errors \
build_pip_pkg

bazel-bin/build_pip_pkg artifacts "--plat-name macosx_11_0_arm64 $NIGHTLY_FLAG"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to manually set the correct wheel name, since we are cross compiling on an x86 machine.

delocate-wheel -w wheelhouse artifacts/*.whl

0 .github/workflows/make_wheel_macOS.sh → .github/workflows/make_wheel_macOS_x86.sh
View file
Open in desktop
File renamed without changes.
27 changes: 24 additions & 3 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,16 @@ jobs:
os: ['macos-latest', 'windows-latest', 'ubuntu-18.04']
py-version: ['3.6', '3.7', '3.8', '3.9']
tf-version: ['2.4.2', '2.6.0']
cpu: ['x86']
include:
- os: 'macos-11'
cpu: 'arm64'
tf-version: '2.5.0'
py-version: '3.8'
- os: 'macos-11'
cpu: 'arm64'
tf-version: '2.5.0'
py-version: '3.9'
Comment on lines +46 to +53
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tensorflow-macos currently only supports TF 2.5.0 and Python 3.8 and 3.9.

Copy link
Contributor

@bhack bhack Sep 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And tensorflow-metal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell tensorflow-metal is an optional dependency to enable metal support via the new pluggable device mechanism so it is not really necessary here since as far as I know TFA doesn't directly link against the metal plugin.

Copy link
Contributor

@bhack bhack Sep 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I meant if It Is constrained to 2.5.0 now I suppose that all the macos stuffs are aligned on 2.5.0 including GPU acell

Copy link
Contributor Author

@lgeiger lgeiger Sep 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, tensorflow-metal is also only built against 2.5.0 currently.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as I told you it is not about conflicts, It is about that if tensorflow-macos is imminent to land on 2.6.0 as tensorflow-deps is ready for TF 2.6.0 I will just wait to edit this PR directly to have a 2.6.0 wheel published for TF addons.
If not probably we will merge as is.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as I told you it is not about conflicts, It is about that if tensorflow-macos is imminent to land on 2.6.0 as tensorflow-deps is ready for TF 2.6.0 I will just wait to edit this PR directly to have a 2.6.0 wheel published for TF addons.
If not probably we will merge as is.

@mihaimaruseac and @penpornk

Thanks @bhack for the heads-up. Is this PR for enabling ARM64 builds for TF through official build channels which will be hosted on pypi ?

Copy link
Contributor

@bhack bhack Sep 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this PR for enabling ARM64 builds for TF through official build channels which will be hosted on pypi ?

No this is limited for Tensorflow Addons ARM64 builds in the official Tesnroflow Addons pypi so we need to know if we could enable 2.6.0 for tensorflow-macos 2.6.0.

ARM64 builds for TF through official build channels which will be hosted on pypi

For this you really need a brand new PR in the TF own repository over:
(Stable)
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/ci_build/rel/macos
(Eventually nightly)
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/ci_build/nightly_release/macos

But it is another topic.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bhack , thanks for the update. We are working on the v2.6 release, I will update here regarding that. I see this PR has merged we can adopt to tensorflow-macos after we make the release in a separate PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kulinseth Are you interested to manage the TF addons wheels release on your pypi on a new tensorflow-macos/metal release?

fail-fast: false
steps:
- uses: actions/[email protected]
Expand Down Expand Up @@ -68,12 +78,13 @@ jobs:
PY_VERSION: ${{ matrix.py-version }}
TF_VERSION: ${{ matrix.tf-version }}
NIGHTLY_TIME: ${{ steps.author-date.outputs.result }}
CPU: ${{ matrix.cpu }}
shell: bash
run: bash .github/workflows/make_wheel_${OS}.sh
run: bash .github/workflows/make_wheel_${OS}_${CPU}.sh
- if: matrix.py-version != '3.9' || matrix.tf-version != '2.4.2'
uses: actions/upload-artifact@v1
with:
name: ${{ runner.os }}-${{ matrix.py-version }}-tf${{ matrix.tf-version }}-wheel
name: ${{ runner.os }}-${{ matrix.py-version }}-tf${{ matrix.tf-version }}-${{ matrix.cpu }}-wheel
path: wheelhouse
upload-wheels:
name: Publish wheels to PyPi
Expand All @@ -84,12 +95,22 @@ jobs:
os: ['macOS', 'Windows', 'Linux']
py-version: ['3.6', '3.7', '3.8', '3.9']
tf-version: ['2.6.0']
cpu: ['x86']
include:
- os: 'macos-11'
cpu: 'arm64'
tf-version: '2.5.0'
py-version: '3.8'
- os: 'macos-11'
cpu: 'arm64'
tf-version: '2.5.0'
py-version: '3.9'
fail-fast: false
if: (github.event_name == 'push' && github.ref == 'refs/heads/master') || github.event_name == 'release'
steps:
- uses: actions/download-artifact@v1
with:
name: ${{ matrix.os }}-${{ matrix.py-version }}-tf${{ matrix.tf-version }}-wheel
name: ${{ matrix.os }}-${{ matrix.py-version }}-tf${{ matrix.tf-version }}-${{ matrix.cpu }}-wheel
path: ./dist
- run: |
set -e -x
Expand Down