Support for using setup.py for install (including for --requirements)

Author: Jonas Thiem

.travis.yml

@@ -1,13 +1,16 @@
 sudo: required
 
+dist: xenial  # needed for more recent python 3 and python3-venv
+
 language: generic
 
 services:
   - docker
 
 before_install:
   - travis_retry sudo apt update -qq
-  - travis_retry sudo apt install -qq --no-install-recommends python2.7 python3
+  - travis_retry sudo apt install -qq --no-install-recommends python2.7 python3 python3-venv python3-virtualenv
+  # (venv/virtualenv are both used by tests/test_pythonpackage.py)
   - sudo pip install tox>=2.0
   # https://github.com/travis-ci/travis-ci/issues/6069#issuecomment-266546552
   - git remote set-branches --add origin master
@@ -28,7 +31,8 @@ env:
 
 before_script:
   # we want to fail fast on tox errors without having to `docker build` first
-  - tox
+  - tox -- tests/ --ignore tests/test_pythonpackage.py
+  # (we ignore test_pythonpackage.py since these run way too long!! test_pythonpackage_basic.py will still be run.)
 
 script:
   - docker build --tag=p4a --file Dockerfile.py3 .

Dockerfile.py3

@@ -95,7 +95,8 @@ ENV WORK_DIR="${HOME_DIR}" \
 
 # install system dependencies
 RUN ${RETRY} apt -y install -qq --no-install-recommends \
-        python3 virtualenv python3-pip wget lbzip2 patch sudo \
+        python3 virtualenv python3-pip python3-venv \
+        wget lbzip2 patch sudo \
     && apt -y autoremove
 
 # build dependencies

doc/source/distutils.rst

@@ -2,8 +2,71 @@
 distutils/setuptools integration
 ================================
 
-Instead of running p4a via the command line, you can integrate with
-distutils and setup.py.
+Have `p4a apk` run setup.py (replaces ``--requirements``)
+---------------------------------------------------------
+
+If your project has a `setup.py` file, then it can be executed by
+`p4a` when your app is packaged such that your app properly ends up
+in the packaged site-packages. (Use ``--use-setup-py`` to enable this,
+``--ignore-setup-py`` to prevent it)
+
+This is functionality to run **setup.py INSIDE `p4a apk`,** as opposed
+to the other section below, which is about running
+*p4a inside setup.py*.
+
+This however has these caveats:
+
+- **Only your ``main.py`` from your app's ``--private`` data is copied
+  into the .apk!** Everything else needs to be installed by your
+  ``setup.py`` into the site-packages, or it won't be packaged.
+
+- All dependencies that map to recipes can only be pinned to exact
+  versions, all other constraints will either just plain not work
+  or even cause build errors. (Sorry, our internal processing is
+  just not smart enough to honor them properly at this point)
+
+- If you don't use Python 3 per default, you still need to specify
+  ``--requirements python2`` (without any additional dependencies)
+
+- The dependency analysis at the start may be quite slow and delay
+  your build
+
+Reasons why you would want to use a `setup.py` to be processed (and
+omit specifying ``--requirements``):
+
+- You want to use a more standard mechanism to specify dependencies
+  instead of ``--requirements``
+
+- You already use a `setup.py` for other platforms
+
+- Your application imports itself
+  in a way that won't work unless installed to site-packages)
+
+
+Reasons **not** to use a `setup.py` (that is to use the usual
+``--requirements`` mechanism instead):
+
+- You don't use a `setup.py` yet, and prefer the simplicity of
+  just specifying ``--requirements``
+
+- Your `setup.py` assumes a desktop platform and pulls in
+  Android-incompatible dependencies, and you are not willing
+  to change this, or you want to keep it separate from Android
+  deployment for other organizational reasons
+
+- You need data files to be around that aren't installed by
+  your `setup.py` into the site-packages folder
+
+
+Use your setup.py to call p4a
+-----------------------------
+
+Instead of running p4a via the command line, you can call it via
+`setup.py` instead, by it integrating with distutils and setup.py.
+
+This is functionality to run **p4a INSIDE setup.py,** as opposed
+to the other section above, which is about running
+*setup.py inside `p4a apk`*.
 
 The base command is::
 
@@ -44,7 +107,7 @@ All of these automatic arguments can be overridden by passing them manually on t
     python setup.py apk --name="Testapp Setup" --version=2.5
 
 Adding p4a arguments in setup.py
---------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Instead of providing extra arguments on the command line, you can
 store them in setup.py by passing the ``options`` parameter to
@@ -79,7 +142,7 @@ setup.py apk``. Any options passed on the command line will override
 these values.
 
 Adding p4a arguments in setup.cfg
----------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 You can also provide p4a arguments in the setup.cfg file, as normal
 for distutils. The syntax is::

doc/source/quickstart.rst

@@ -298,6 +298,14 @@ your own Kivy branch you might set::
 The specified directory will be copied into python-for-android instead
 of downloading from the normal url specified in the recipe.
 
+setup.py file (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If your application is also packaged for desktop using `setup.py`,
+you may want to use your `setup.py` instead of the
+``--requirements`` option to avoid specifying things twice.
+For that purpose, check out :doc:`distutils`
+
 Going further
 ~~~~~~~~~~~~~
 

pythonforandroid/bootstraps/common/build/build.py

@@ -23,12 +23,14 @@
 import jinja2
 
 
-def get_dist_info_for(key):
+def get_dist_info_for(key, error_if_missing=True):
     try:
         with open(join(dirname(__file__), 'dist_info.json'), 'r') as fileh:
             info = json.load(fileh)
-        value = str(info[key])
+        value = info[key]
     except (OSError, KeyError) as e:
+        if not error_if_missing:
+            return None
         print("BUILD FAILURE: Couldn't extract the key `" + key + "` " +
               "from dist_info.json: " + str(e))
         sys.exit(1)
@@ -304,18 +306,45 @@ def make_package(args):
         f.write("P4A_MINSDK=" + str(args.min_sdk_version) + "\n")
 
     # Package up the private data (public not supported).
+    use_setup_py = get_dist_info_for("use_setup_py",
+                                     error_if_missing=False) is True
     tar_dirs = [env_vars_tarpath]
-    if args.private:
-        tar_dirs.append(args.private)
-    for python_bundle_dir in ('private', 'crystax_python', '_python_bundle'):
-        if exists(python_bundle_dir):
-            tar_dirs.append(python_bundle_dir)
-    if get_bootstrap_name() == "webview":
-        tar_dirs.append('webview_includes')
-    if args.private or args.launcher:
-        make_tar(
-            join(assets_dir, 'private.mp3'), tar_dirs, args.ignore_path,
-            optimize_python=args.optimize_python)
+    _temp_dirs_to_clean = []
+    try:
+        if args.private:
+            if not use_setup_py or (
+                    not exists(join(args.private, "setup.py")) and
+                    not exists(join(args.private, "pyproject.toml"))
+                    ):
+                print('No setup.py/pyproject.toml used, copying '
+                      'full private data into .apk.')
+                tar_dirs.append(args.private)
+            else:
+                print('Copying main.py ONLY, since other app data is '
+                      'expected in site-packages.')
+                main_py_only_dir = tempfile.mkdtemp()
+                _temp_dirs_to_clean.append(main_py_only_dir)
+                if exists(join(args.private, "main.pyo")):
+                    shutil.copyfile(join(args.private, "main.pyo"),
+                                    join(main_py_only_dir, "main.pyo"))
+                elif exists(join(args.private, "main.py")):
+                    shutil.copyfile(join(args.private, "main.py"),
+                                    join(main_py_only_dir, "main.py"))
+                tar_dirs.append(main_py_only_dir)
+        for python_bundle_dir in ('private',
+                                  'crystax_python',
+                                  '_python_bundle'):
+            if exists(python_bundle_dir):
+                tar_dirs.append(python_bundle_dir)
+        if get_bootstrap_name() == "webview":
+            tar_dirs.append('webview_includes')
+        if args.private or args.launcher:
+            make_tar(
+                join(assets_dir, 'private.mp3'), tar_dirs, args.ignore_path,
+                optimize_python=args.optimize_python)
+    finally:
+        for directory in _temp_dirs_to_clean:
+            shutil.rmtree(directory)
 
     # Remove extra env vars tar-able directory:
     shutil.rmtree(env_vars_tarpath)
@@ -361,9 +390,7 @@ def make_package(args):
     version_code = 0
     if not args.numeric_version:
         # Set version code in format (arch-minsdk-app_version)
-        with open(join(dirname(__file__), 'dist_info.json'), 'r') as dist_info:
-            dist_data = json.load(dist_info)
-        arch = dist_data["archs"][0]
+        arch = get_dist_info_for("archs")[0]
         arch_dict = {"x86_64": "9", "arm64-v8a": "8", "armeabi-v7a": "7", "x86": "6"}
         arch_code = arch_dict.get(arch, '1')
         min_sdk = args.min_sdk_version