Use environment variable for switching SoX dep (#669)

* Use env var for switching SoX build and default to not

* Update README

* Fix packaging/CI script

Author: mthrok

.circleci/unittest/linux/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-python setup.py develop
+BUILD_SOX=1 python setup.py develop

.circleci/unittest/windows/scripts/install.sh

@@ -20,4 +20,4 @@ printf "Installing PyTorch with %s\n" "${cudatoolkit}"
 conda install -y -c pytorch-nightly pytorch "${cudatoolkit}"
 
 printf "* Installing torchaudio\n"
-IS_CONDA=true python setup.py develop
+python setup.py develop

README.md

@@ -27,29 +27,16 @@ to use and feel like a natural extension.
 Dependencies
 ------------
 * pytorch (nightly version needed for development)
-* libsox v14.3.2 or above
+* libsox v14.3.2 or above (only required when building from source)
 * [optional] vesis84/kaldi-io-for-python commit cb46cb1f44318a5d04d4941cf39084c5b021241e or above
 
-Quick install on
-OSX (Homebrew):
-```bash
-brew install sox
-```
-Linux (Ubuntu):
-```bash
-sudo apt-get install sox libsox-dev libsox-fmt-all
-```
-Anaconda
-```bash
-conda install -c conda-forge sox
-```
-
 Installation
 ------------
 
-### Binaries
+### Binary Distibutions
 
 To install the latest version using anaconda, run:
+
 ```
 conda install -c pytorch torchaudio
 ```
@@ -64,26 +51,48 @@ pip install torchaudio -f https://download.pytorch.org/whl/torch_stable.html
 torch from PyPI. If you need a different torch configuration, preinstall torch
 before running this command.)
 
-At the moment, there is no automated nightly build process, but we occasionally
-build nightlies based on PyTorch nightlies by hand following the instructions in
-[packaging](packaging).  To install the latest nightly via pip, run:
+### Nightly build
+
+Note that nightly build is build on PyTorch's nightly build. Therefore, you need to install the latest PyTorch when you use nightly build of torchaudio.
+
+**pip**
 
 ```
 pip install numpy
 pip install --pre torchaudio -f https://download.pytorch.org/whl/nightly/torch_nightly.html
 ```
 
-To install the latest nightly via conda, run:
+**conda**
 
 ```
 conda install -y -c pytorch-nightly torchaudio
 ```
 
-
 ### From Source
 
 If your system configuration is not among the supported configurations
-above, you can build from source.
+above, you can build torchaudio from source.
+
+This will require libsox v14.3.2 or above.
+
+<Details><Summary>Click here for the examples on how to install SoX</Summary>
+
+OSX (Homebrew):
+```bash
+brew install sox
+```
+
+Linux (Ubuntu):
+```bash
+sudo apt-get install sox libsox-dev libsox-fmt-all
+```
+
+Anaconda
+```bash
+conda install -c conda-forge sox
+```
+
+</Details>
 
 ```bash
 # Linux
@@ -93,8 +102,33 @@ python setup.py install
 MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install
 ```
 
-If while building from within an anaconda environment you come across errors similar to the following:
+Alternatively, the build process can build SoX (and codecs such as libmad, lame and flac) statically and torchaudio can link them, by setting environment variable `BUILD_SOX=1`.
+The build process will fetch and build SoX, liblame, libmad, flac before building extension.
 
+```bash
+# Linux
+BUILD_SOX=1 python setup.py install
+
+# OSX
+BUILD_SOX=1 MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install
+```
+
+This is known to work on linux and unix distributions such as Ubuntu and CentOS 7 and macOS.
+If you try this on a new system and found a solution to make it work, feel free to share it by opening and issue.
+
+#### Troubleshooting
+
+<Details><Summary>checking build system type... ./config.guess: unable to guess system type</Summary>
+
+Since the configuration file for codecs are old, they cannot correctly detect the new environments, such as Jetson Aarch. You need to replace the `config.guess` file in `./third_party/tmp/lame-3.99.5/config.guess` and/or `./third_party/tmp/libmad-0.15.1b/config.guess` with [the latest one](https://github.com/gcc-mirror/gcc/blob/master/config.guess).
+
+See also: [#658](https://github.com/pytorch/audio/issues/658)
+
+</Details>
+
+<Details><Summary>Undefined reference to `tgetnum' when using `BUILD_SOX`</Summary>
+
+If while building from within an anaconda environment you come across errors similar to the following:
 
 ```
 ../bin/ld: console.c:(.text+0xc1): undefined reference to `tgetnum'
@@ -107,6 +141,8 @@ Install `ncurses` from `conda-forge` before running `python setup.py install`:
 conda install -c conda-forge ncurses
 ```
 
+</Details>
+
 
 Quick Usage
 -----------

build_tools/setup_helpers/extension.py

@@ -1,3 +1,4 @@
+import os
 import platform
 import subprocess
 from pathlib import Path
@@ -18,9 +19,21 @@
 _TP_BASE_DIR = _ROOT_DIR / 'third_party'
 _TP_INSTALL_DIR = _TP_BASE_DIR / 'build'
 
-# Temporary fix for building in fbcode
-# at the moment, we have to use external sox in fbcode
-_BUILD_DEPS = not (_ROOT_DIR / '.use_external_sox').exists()
+
+def _get_build_sox():
+    val = os.environ.get('BUILD_SOX', '0')
+    trues = ['1', 'true', 'TRUE', 'on', 'ON', 'yes', 'YES']
+    falses = ['0', 'false', 'FALSE', 'off', 'OFF', 'no', 'NO']
+    if val in trues:
+        return True
+    if val not in falses:
+        print(
+            f'WARNING: Unexpected environment variable value `BUILD_SOX={val}`. '
+            f'Expected one of {trues + falses}')
+    return False
+
+
+_BUILD_SOX = _get_build_sox()
 
 
 def _get_eca(debug):
@@ -52,14 +65,14 @@ def _get_include_dirs():
     dirs = [
         str(_ROOT_DIR),
     ]
-    if _BUILD_DEPS:
+    if _BUILD_SOX:
         dirs.append(str(_TP_INSTALL_DIR / 'include'))
     return dirs
 
 
 def _get_extra_objects():
     objs = []
-    if _BUILD_DEPS:
+    if _BUILD_SOX:
         # NOTE: The order of the library listed bellow matters.
         #
         # (the most important thing is that dependencies come after a library
@@ -71,7 +84,7 @@ def _get_extra_objects():
 
 
 def _get_libraries():
-    return [] if _BUILD_DEPS else ['sox']
+    return [] if _BUILD_SOX else ['sox']
 
 
 def _build_codecs():
@@ -106,6 +119,6 @@ def get_ext_modules(debug=False):
 
 class BuildExtension(TorchBuildExtension):
     def build_extension(self, ext):
-        if ext.name == _EXT_NAME and _BUILD_DEPS:
+        if ext.name == _EXT_NAME and _BUILD_SOX:
             _configure_third_party()
         super().build_extension(ext)

packaging/build_wheel.sh

@@ -15,5 +15,5 @@ if [[ "$OSTYPE" == "msys" ]]; then
     python_tag="$(echo "cp$PYTHON_VERSION" | tr -d '.')"
     python setup.py bdist_wheel --plat-name win_amd64 --python-tag $python_tag
 else
-    python setup.py bdist_wheel
+    BUILD_SOX=1 python setup.py bdist_wheel
 fi

packaging/torchaudio/build.sh

@@ -1,4 +1,4 @@
 #!/usr/bin/env bash
 set -ex
 
-python setup.py install --single-version-externally-managed --record=record.txt
+BUILD_SOX=1 python setup.py install --single-version-externally-managed --record=record.txt