diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt new file mode 100644 index 0000000..223683f --- /dev/null +++ b/pep-0497-externals.txt @@ -0,0 +1,460 @@ +PEP: 497 +Title: External dependencies +Version: $Revision$ +Last-Modified: $Date$ +Author: Tennessee Leeuwenburg , + +Status: Draft +Type: Standards Track +Python-Version 3.6 +Content-Type: text/x-rst +Created: 03-Aug-2015 +Post-History: 03-Aug-2015 + +Very Short Abstract +=================== + +This PEP covers three things. + +1. How users can specify external dependencies in setup.py +2. How users can specify external dependencies in requirements.txt +3. How external dependencies will be communicated between tools, + (setuptools --> wheel, wheel --> pip, ...) + +It recommends additional syntax to specify the requirements, implementation +recommendations on how to provide appropriate messages to users, and relevant +detailed discussion on technical issues. Where only setup.py is referenced, +the scope should be taken to include requirements.txt unless otherwise +mentioned. + +Abstract +======== + +The canonical method for installing Python packages is to use pip and/or +setuptools, whether that is referencing PyPI or a local repository. A limitation +of the current install ecosystem is the relationship with non-Python +dependencies. This can result in compilation failures or execution failures. +Package maintainers may wish to specify the requirements to avoid this issue, +and users may appreciate earlier and more informative messages about their +system. + +The responsibility of pip is to stop early and provide useful information about +missing (and not automatically installable) dependencies. Ideally it would provide +a useful user message about how to obtain the required dependencies. The +responsibility for actual installation lies with the user at this time. + +This PEP proposes a mechanism for extending the requirements section of +setup.py, and the content of requirements.txt, which will allow authors to +express non-Python requirements cleanly. In the implementation section, a +discussion of how this may be used to solve problems is given. + +Here is an example of the proposed syntax for setup.py, where both operating +system and build target specifiers are included. This is "Concept 1" of the +alternatives discussed later in the document. The syntax for requirements.txt +is similar, and is included later in the document. + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires=['yaml', 'scipy[atlas]', 'lib!libblas1.so'], + setup_requires=[ + 'pyfortran', + 'bin!fortran:dist_name=="ubuntu"', + 'include!kernel.h', + 'include!liblas.h', + ], + ) + +If the user does not have the required dependencies, the user will be presented +with a useful install message and a process that exits early, as opposed to +relying on the compiler to fail and provide an error message that may not be +readily understood by most users. + +The mechanism of referencing failed dependencies such as 'kernel.h' to useful +user messages regarding the actual operating system packages that are relevant +is discussed in the implementation section, along with the rationale for +handling that referencing outside of setup.py. + +Rationale +========= + +Differences between environments have implications for the building, testing, +installing and execution of software. There are two sides to this coin. First, +in some environments, additional Python packages are needed. Secondly, in some +environments, additional non-Python packages need to be present, such as header +files, libraries, scripts or other binaries. The syntax proposed here allows +the description of both sorts of requirements in a single syntax. + +Setuptools and pip should be modified to support the use case of installing +the appropriate python packages as required depending on the specification. + +They should also support aborting an install with a useful error message if +required dependecies are not installed in the environment. The responsibility +for actually installing those required packages lies with the user. + +This would allow automatic installation of environment-specific Python +dependencies and pave the way to improving how users go about installing +non-Python prerequisites. + +Ideally, any error messages will include the package manager command required +to add the missing environment dependencies. + +It is expected that this feature would be commonly used, particularly in +scientific applications which frequently have complex interactions with +externally provided libraries. The example provided in the abstract is a +hypothetical package which needs versions of numpy and scipy, both of which +must have been compiled to be aware of the ATLAS compiled set of linear algebra +libraries (for performance reasons). This sounds esoteric but is, in fact, a +routinely encountered situation which drives people towards using the +alternative packaging for scientific python environments. + +Syntax +====== + +The proposed syntax is an external object type, an "!", and then an external +object name. The object type can be one of: + + include + bin + lib + +The object name should be the name of the required thing - search paths are +presumed to be used to find the object. + +These dependencies may be controlled via environment markers in the standard +fashion (see PEP 496). + +For example, consider the line + + setup_requires=['bin!fortran:dist_name=="ubuntu"'] + +The 'bin' specifies one of "bin", "lib" or "header", representing the only +non-Python dependency types allows. The absence of this prefix implies the the +package name is a discoverable Python package. This rule is controlled via the +environment marker syntax to apply only on the Ubuntu distribution. + +Implementation Concerns +======================= + +Short Version +------------- + +1. Setuptool + pip should display relevant information earlier than now +2. A mechanism to assist in identifying external package names would be useful +3. The maintenance burden for (2) should fall on the package management system + maintainers, not on pip/setuptools developers +4. This means a simple pluging approach to mapping from specific resources + to relevant external package names and invocation calls +5. I propose this looks like "pip install vendor_magic darwin" +6. And that the various O/Ses ship this pre-installed in their distributions + +Platform Abstraction +-------------------- + +Pip and setuptools cannot (and should not) be expected to provide detailed +reasoning about the external environment and appropriate responses. Questions +have arisen often about the degree to which the Python ecosystem build tools +can or should auto-detect or otherwise handle the collection of external +dependencies at build time. + +Extending the Python build and packaging system into the external environment +space multiplies the responsibility of package authors excessively. The entities +which are appropriately informed and resourced to handle the complexities +of complex build needs are the environment tool maintainers. + +The issue that is being solved is a reasonable communications layer by which +information can be shared between those two separated ecosystems. Python +package maintainers who are aware they are building a system-dependent package +can take steps to represent known critical dependencies in their configurations +so that developers can discover these more easily when building or installing +tools. This is, in part, due to the fact that standard build tools often fail +late in the build process and themselves provide less information that is ideal +to end users. + +Given the broad use of Python by many users, providing additional information +here is extremely useful. Further, these tools provide the means for external +package systems to respond appropriately to the dependency needs of Python +packages. + + +Setuptools +---------- + +Wheel +----- + +Pip +--- + +Case sensitivity of specifiers +------------------------------ + +Hooks, stages and processes +--------------------------- + +1. Perform checks of availability of resources +2. Provide a callback hook to external tools. These hooks can return either + failure (no recourse) or success (resource provided). In this way, it would + be possible to support logging (returns failure but has a useful side-effect) + or a problem-resolution response (actually installs the external dependency) +3. If there are remaining unresolved dependencies, proceed to an early exit +4. Provide a callback hook indicating early exit along with appropriate + messaging indicating the unresolved external dependencies +5. Provide final messages to the user + +The vendor magic layer would be implemented as a stage four callback tool which +was capable of understanding the resource specifications and messaging, and +supplement the standard pip/setuptools messaging with appropriate information +for appropriate packages to install and/or suitable actions the user could take. + +Discussion of Implementation Concerns +------------------------------------- + +A common user experience is to attempt the installation of software in a +basic environment, and for the installation to fail during the compilation +of a dependency. The user is presented with an error message provided by the +compiler. Not all users have the ability to understand these messages, and +a common experience is to spend time searching online for the appopriate +package which is provided by the vendor. + +The base implementation will also only provide information about missing +headers, libraries or executables. This is a step forward in that the error +message is provided very early in the install process, rather than potentially +many minutes in to a large compile job. + +A key aspect of the usability of this tool is a mechanism to provide more useful +information to the user about how to obtain those missing resources. +The proposal is to provide a layer, consumed by pip, which is capable of +providing information about appropriate package names. It is surprising that +vendors do not already provide a good tool for this, given that finding +appropriate header and library files is a common experience in the general +linux ecosystem. However, it is also a critical aspect of the user experience, +and something which package maintainers will need to be able to specify +precisely. The proposed approach would be for pip to supply the ability to +install an inference layer, usable by users and vendors alike. Usage would be +along the lines of + + pip install vendor_magic darwin + +This would transform the user messages relating to missing dependencies from + + Could not find libatlas.so + +to, e.g., + + Could not find libatlas.so. Run 'port install atlas' + + +Discussion / FAQ / Issues +========================= + +This section will be updated to capture incoming feedback and represents a +snapshot of the current concerns. + + +Specification of files (headers, libs, bins) vs packages +-------------------------------------------------------- + +There was a discussion about whether specific dependencies (named headers, +libraries, binaries etc) should be included, or whether general package names +(such as may be expected by yum, apt, conda) were appropriate. Due to the +number of package management tools, plus the changing nature of those, it +was decided that concentrating on the specific files was more useful than +using package names. It also provides looser coupling between the specification +of externals and the package management tools. + +Location in setup.py rather than requirements.txt +------------------------------------------------- + +Requirements.txt is scoped to the use for local deployments to specify +requirements precisely for the relevant local environment. Setup.py is the +appropriate place to specify the information for the package in general. + +In future, there may be some scope for including similar directives within +requirements.txt, but that is out of scope for this PEP. + +Prior Approaches +---------------- + +Some capacity to support description of external dependencies (among other +things) was described in PEPs 314 and 345 (Metadata 1.1 and 1.2). These +describe directives whichs can be placed in the PKG-INFO file. An implementation +responding to these directives is pending. + +This PEP places the directives in setup.py. This approach is closer to the +typical approach of package maintainers to expressing requirements and places +the external dependency specification nearer to the Python dependency +specification. This PEP would supercede PEPs 314 and 345 with respect to +external dependencies. + +Addition of vendor inference layer +---------------------------------- + +Bearing in mind the discussion about using specific dependencies, the ideal +user experience is to get a useful message about how to get the package +installed on their system. This means providing a message which is relevant +for their package management environment (e.g. yum, apt, conda). Each +distributor would be expected to provide an inference lookup which can be +installed into pip which would assist with identifying the packages which +are associated with the named dependencies. + +Implementation of the vendor inference layer +-------------------------------------------- + +There may be some advantage to regard the vendor inference layer as TBD in +a future PEP due to the complexity and also in order to better focus the scope +of the PEP. This externals PEP can proceed effectively without the vendor +inference layer being implemented or specified. Alternatively, a separate PEP +could be developed in tandem with this proposal to hold the apppropriate level +of detail. + + +Types of environment and vendor +------------------------------- + +Anaconda is not an operating system. Neither is yum, neither is apt. It is the +maintainer of the packaging system which is pertinent, and the current operating +system is an imperfect indicator of the relevant package manager. + + +Directories, categories and locations of dependency files +--------------------------------------------------------- + +The proposed mechanism allows introspection of the standard location for +packages. On *nix environments, these would be (for example), /usr/bin, +/usr/local/include etc (see https://www.gnu.org/prep/standards/html_node/Directory-Variables.html +for a discussion of standard directories). Non-standard dependency locations +would be supported only via the underlying platform's mechanisms for exposing +these details, such as PATH, LD_LIBRARY_PATH etc. This PEP does not include +generalised configuration and specification of dependencies in non-standard +or customised locations. + +The Python installation tools should honour those standard mechanisms. +The details are per-platform and not specified by this specification +(see Platform Abstraction for more on this point). + +Virtual Environment Concerns +---------------------------- + +This should work fine with virtual environments. Since the PEP draws the line +at performing external package installation, it is up to package management +tools to decide how to separate concerns between systemwide installation +and virtual environment installation. For example, a system such as +Anaconda which is heavily based around virtual environments is well capable +of handling the installation of external dependencies into a virtual environment +context. + +Alternative Syntax Concepts +========================== + +Concept 0: Separated external and non-external dependencies +----------------------------------------------------------- + +This is an early-stage syntax. Concept 1 replaces it as the current +"front runner" as implementation candidate and is included in the abstract. + +The advantage of Concept 0 is that is clearly separates the non-Python +dependency specification into an isolated part of the configuration arguments, +reducing the potential for confusion between external and internal dependencies. +This may assist during implementation as no 'sniffer' will be needed to +determine whether a named package is internal or external. + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires='yaml, scipy[atlas]', + setup_requires='pyfortran', + external={ + setup=[ + 'bin!fortran:dist_name=='ubuntu', + 'include!kernel.h', + 'include!liblas.h', + ], + install=[ + 'lib!liblas1.so' + ], + } + ) + +Concept 1: Integrate external and non-external dependencies +----------------------------------------------------------- + +A good argument for integrating these is that it will allow extensions +to requirements.txt without needing a separate file to exist for the externals. +It is shorter and still highly readable. Potentially instead of e.g. "lib!" +for a prefix, "extlib!" could be used to signpost more strongly. But it's +longer. + +Another advantage to this approach it that any tool which treats setup_requires +(et al) as a list of opaque strings should "just work", while a separate field +causes problems. + +Notably, this change should mostly be forwards compatible with the overall +structure of PEP 426. + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires=['yaml', 'scipy[atlas]', 'lib!libblas1.so'], + setup_requires=[ + 'pyfortran', + 'bin!fortran:dist_name=="ubuntu"', + 'include!kernel.h', + 'include!liblas.h', + ], + ) + + +Concept 2: Python-like language +------------------------------- + +This isn't really valid code and is included mainly to document the development +of ideas. It illustrates an alternative structure to using separate config +areas for setup, install etc and treats those as a part of the trigger +for a particular dependency, called the target. There is still some advantage +to the clarity and flexibility provided by this approach. + +:: + + setup( + install_requires='yaml', + setup_requires='pyfortran', + os_requires=[ + ['fortran', type='bin', os='ubuntu', 'target=build'], + ['kernel.h', type='header', os='linux', target='build'], + ['libblas.o', type='lib', target='run'] + ['scipy +atlas', type='packagename', target='run', os='darwin'] + ] + ) + + +References +========== + +.. [1] PEP 426, Metadata for Python Software Packages, Coghlan, Holth, Stufft + (http://www.python.org/dev/peps/pep-0426) + +.. [2] PEP 345, Sample Plaintext PEP Template, Warsaw + (http://www.python.org/dev/peps/pep-0009) + +.. [3] Pull request 29 on interoperability-peps + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: