Building Dart SDK fails at `fetch dart` on Apple Silicon / DTK / macOS 11

[alexmarkley]

This is almost certainly not an issue in the Dart SDK proper, but an upstream tooling issue. That said I don't have a better place to report this, so here we go...

Steps to Reproduce
Follow these instruction on building the Dart SDK: https://github.com/dart-lang/sdk/wiki/Building

• Install depot_tools
• do fetch dart

Observed Behavior
fetch dart cannot download the Dart SDK source code as per the documentation.

Expected Behavior
I would like to be able to download the Dart SDK source code.

Runtime Environment

DTK:dart-sdk alex$ uname -a
Darwin DTK.mylocal 20.1.0 Darwin Kernel Version 20.1.0: Sat Sep 12 20:05:59 PDT 2020; root:xnu-7195.40.84.171.4~1/RELEASE_ARM64_T8020 arm64
DTK:dart-sdk alex$ which python
/usr/bin/python
DTK:dart-sdk alex$ python --version
Python 2.7.16
DTK:dart-sdk alex$ 

Notes on Python 2
As per the comment at https://github.com/dart-lang/sdk/wiki/Building#python-2 I did also try installing virtualenv and creating a python2 virtualenv prior to fetching the SDK. This had no discernible effect, since the system default python is already python 2.7.

Why am I trying to do this?
I realize that Dart is available to download for many architectures without compiling from source. I want a native (preferably "universal") toolchain for my Flutter iOS & macOS development purposes, with the ultimate goal of targeting universal macOS desktop binaries for my Flutter app. I am interested in helping in the porting effort by reporting issues I encounter, testing experimental builds, and even writing some patches if I can.

Log

DTK:Build alex$ which fetch
/Users/alex/Build/depot_tools/fetch
DTK:Build alex$ mkdir dart-sdk
DTK:Build alex$ cd dart-sdk
DTK:dart-sdk alex$ fetch dart
Running: gclient root
WARNING: Your metrics.cfg file was invalid or nonexistent. A new one will be created.
Running: gclient config --spec 'solutions = [
  {
    "name": "sdk",
    "url": "https://dart.googlesource.com/sdk.git",
    "deps_file": "DEPS",
    "managed": False,
    "custom_deps": {},
  },
]
'
Running: gclient sync

________ running 'arch -arch arm64e -arch arm64 git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://dart.googlesource.com/sdk.git /Users/alex/Build/dart-sdk/_gclient_sdk_4o5a0hy_' in '/Users/alex/Build/dart-sdk'
Cloning into '/Users/alex/Build/dart-sdk/_gclient_sdk_4o5a0hy_'...
remote: Sending approximately 1.05 GiB ...
remote: Counting objects: 1475, done
remote: Finding sources: 100% (34/34)
remote: Total 1044242 (delta 798152), reused 1044231 (delta 798152)
Receiving objects: 100% (1044242/1044242), 1.04 GiB | 47.34 MiB/s, done.
Resolving deltas: 100% (798152/798152), done.
Syncing projects: 100% (100/100), done.                                                                   

________ running 'download_from_google_storage --no_auth --no_resume --bucket dart-dependencies --recursive --auto_platform --extract --directory sdk/third_party/firefox_jsshell' in '/Users/alex/Build/dart-sdk'
0> Failed to fetch file gs://dart-dependencies/6cc132344cae69172a69cca2ff503da84e5b9c6c for sdk/third_party/firefox_jsshell/mac/jsshell.tar.gz, skipping. [Err: [E2020-09-18T20:15:21.743658-04:00 49300 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/alex/.vpython-root/f31a96]
Process output:
New python executable in /Users/alex/.vpython-root/f31a96/bin/python
ERROR: The executable /Users/alex/.vpython-root/f31a96/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0' (should be u'/Users/alex/.vpython-root/f31a96')
ERROR: virtualenv is not compatible with this system or executable

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
HOME=/tmp/vpython_bootstrap972716080
LANG=en_US.UTF-8
LOGNAME=alex
PATH=/Users/alex/Build/depot_tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/alex/Build/depot_tools:/Users/alex/Build/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/alex/Build/dart-sdk
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHELL=/bin/bash
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.hNHXObY3dW/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=439
TERM_SESSION_ID=DCF501EB-BE65-4402-BEB1-EBE63589ED69
TMP=/tmp
TMPDIR=/tmp
USER=alex
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/alex/.vpython-root/56f65e
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.apple.Terminal
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
[E2020-09-18T20:15:21.748188-04:00 49300 0 annotate.go:241] original error: exit status 100
[E2020-09-18T20:15:21.748269-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748282-04:00 49300 0 annotate.go:241] goroutine 1:
[E2020-09-18T20:15:21.748293-04:00 49300 0 annotate.go:241] #0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
[E2020-09-18T20:15:21.748300-04:00 49300 0 annotate.go:241]   reason: failed to create VirtualEnv
[E2020-09-18T20:15:21.748313-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748319-04:00 49300 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
[E2020-09-18T20:15:21.748334-04:00 49300 0 annotate.go:241]   reason: failed to install VirtualEnv
[E2020-09-18T20:15:21.748341-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748349-04:00 49300 0 annotate.go:241] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2020-09-18T20:15:21.748355-04:00 49300 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2020-09-18T20:15:21.748381-04:00 49300 0 annotate.go:241] #4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
[E2020-09-18T20:15:21.748390-04:00 49300 0 annotate.go:241] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2020-09-18T20:15:21.748396-04:00 49300 0 annotate.go:241]   reason: failed to create new VirtualEnv
[E2020-09-18T20:15:21.748403-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748412-04:00 49300 0 annotate.go:241] #6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
[E2020-09-18T20:15:21.748423-04:00 49300 0 annotate.go:241] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2020-09-18T20:15:21.748430-04:00 49300 0 annotate.go:241] #8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
[E2020-09-18T20:15:21.748443-04:00 49300 0 annotate.go:241]   reason: failed to create empty probe environment
[E2020-09-18T20:15:21.748451-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748458-04:00 49300 0 annotate.go:241] #9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2020-09-18T20:15:21.748464-04:00 49300 0 annotate.go:241] #10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2020-09-18T20:15:21.748477-04:00 49300 0 annotate.go:241] #11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2020-09-18T20:15:21.748484-04:00 49300 0 annotate.go:241] #12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2020-09-18T20:15:21.748492-04:00 49300 0 annotate.go:241] #13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2020-09-18T20:15:21.748499-04:00 49300 0 annotate.go:241] #14 vpython/main.go:106 - main.mainImpl()
[E2020-09-18T20:15:21.748514-04:00 49300 0 annotate.go:241] #15 vpython/main.go:112 - main.main()
[E2020-09-18T20:15:21.748520-04:00 49300 0 annotate.go:241] #16 runtime/proc.go:204 - runtime.main()
[E2020-09-18T20:15:21.748528-04:00 49300 0 annotate.go:241] #17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Failed to fetch file gs://dart-dependencies/6cc132344cae69172a69cca2ff503da84e5b9c6c for sdk/third_party/firefox_jsshell/mac/jsshell.tar.gz. [Err: [E2020-09-18T20:15:21.743658-04:00 49300 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/alex/.vpython-root/f31a96]
Process output:
New python executable in /Users/alex/.vpython-root/f31a96/bin/python
ERROR: The executable /Users/alex/.vpython-root/f31a96/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0' (should be u'/Users/alex/.vpython-root/f31a96')
ERROR: virtualenv is not compatible with this system or executable

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
HOME=/tmp/vpython_bootstrap972716080
LANG=en_US.UTF-8
LOGNAME=alex
PATH=/Users/alex/Build/depot_tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/alex/Build/depot_tools:/Users/alex/Build/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/alex/Build/dart-sdk
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHELL=/bin/bash
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.hNHXObY3dW/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=439
TERM_SESSION_ID=DCF501EB-BE65-4402-BEB1-EBE63589ED69
TMP=/tmp
TMPDIR=/tmp
USER=alex
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/alex/.vpython-root/56f65e
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.apple.Terminal
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
[E2020-09-18T20:15:21.748188-04:00 49300 0 annotate.go:241] original error: exit status 100
[E2020-09-18T20:15:21.748269-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748282-04:00 49300 0 annotate.go:241] goroutine 1:
[E2020-09-18T20:15:21.748293-04:00 49300 0 annotate.go:241] #0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
[E2020-09-18T20:15:21.748300-04:00 49300 0 annotate.go:241]   reason: failed to create VirtualEnv
[E2020-09-18T20:15:21.748313-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748319-04:00 49300 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
[E2020-09-18T20:15:21.748334-04:00 49300 0 annotate.go:241]   reason: failed to install VirtualEnv
[E2020-09-18T20:15:21.748341-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748349-04:00 49300 0 annotate.go:241] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2020-09-18T20:15:21.748355-04:00 49300 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2020-09-18T20:15:21.748381-04:00 49300 0 annotate.go:241] #4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
[E2020-09-18T20:15:21.748390-04:00 49300 0 annotate.go:241] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2020-09-18T20:15:21.748396-04:00 49300 0 annotate.go:241]   reason: failed to create new VirtualEnv
[E2020-09-18T20:15:21.748403-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748412-04:00 49300 0 annotate.go:241] #6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
[E2020-09-18T20:15:21.748423-04:00 49300 0 annotate.go:241] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2020-09-18T20:15:21.748430-04:00 49300 0 annotate.go:241] #8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
[E2020-09-18T20:15:21.748443-04:00 49300 0 annotate.go:241]   reason: failed to create empty probe environment
[E2020-09-18T20:15:21.748451-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748458-04:00 49300 0 annotate.go:241] #9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2020-09-18T20:15:21.748464-04:00 49300 0 annotate.go:241] #10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2020-09-18T20:15:21.748477-04:00 49300 0 annotate.go:241] #11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2020-09-18T20:15:21.748484-04:00 49300 0 annotate.go:241] #12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2020-09-18T20:15:21.748492-04:00 49300 0 annotate.go:241] #13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2020-09-18T20:15:21.748499-04:00 49300 0 annotate.go:241] #14 vpython/main.go:106 - main.mainImpl()
[E2020-09-18T20:15:21.748514-04:00 49300 0 annotate.go:241] #15 vpython/main.go:112 - main.main()
[E2020-09-18T20:15:21.748520-04:00 49300 0 annotate.go:241] #16 runtime/proc.go:204 - runtime.main()
[E2020-09-18T20:15:21.748528-04:00 49300 0 annotate.go:241] #17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Downloading 1 files took 1.046263 second(s)
Error: Command 'download_from_google_storage --no_auth --no_resume --bucket dart-dependencies --recursive --auto_platform --extract --directory sdk/third_party/firefox_jsshell' returned non-zero exit status 1 in /Users/alex/Build/dart-sdk
Subprocess failed with return code 2.
DTK:dart-sdk alex$ 

[alexmarkley]

Bumping this because I'm still unable to download the Dart SDK source code. I'd love to contribute to testing/patching for Apple Silicon as a host, but I'm currently blocked on this issue.

[jmagman]

\cc @a-siva @zanderso where's the best place to track this? If here, who can we poke to take a look?

As @alexmarkley mentioned it's surely not here, but maybe @a-siva knows how to work around it since he has it working? Or maybe the docs needs an update for ARM?

[jmagman]

https://bugs.chromium.org/p/chromium/issues/list ?

[alexmarkley]

@jmagman Thank you, I'll take a look at that issue tracker and see what I can find. A cursory glance seems to show issues being tracked related to macOS 11 on both Intel and ARM, so hopefully there will be traction soon.

I'm also a little concerned because the Dart project explicitly calls out Python 2 as a requirement, and Python 2 is super EOL at this point.

[alexmarkley]

Seems like this might be the most immediately applicable upstream issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1103275&q=component%3AInfra%3ESDK%20macos&can=2

In the issue it seems like partial workarounds are discussed, but I don't know enough about depot_tools to understand the discussion.

[mkustermann]

Making Dart SDK buildable on Apple Sililcon is in-progress. @a-siva is working on this afaik.

[alexmarkley]

Some fixes have landed in MacPorts, and I was able to port install python27 which seems to have improved the above situation.

Now I'm hung up on what appears to be a CPU-bound process internal to depot_tools:

This thing has been running for a couple of days now and I'm not sure how to measure its progress. I'll report upstream as well...

[alexmarkley]

More detailed notes upstream: https://bugs.chromium.org/p/chromium/issues/detail?id=1103275#c10

[Sunbreak]

Stuck here as well. What is 3pp build of cpython in https://bugs.chromium.org/p/chromium/issues/detail?id=1103275#c12?