From 4609351dbc719bed5250748548dcd54f042470c7 Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Mon, 3 Aug 2015 15:35:13 +1000 Subject: [PATCH 1/5] Adding the draft status PEP for external dependency expression --- pep-0497-externals.txt | 252 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 pep-0497-externals.txt diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt new file mode 100644 index 0000000..061319e --- /dev/null +++ b/pep-0497-externals.txt @@ -0,0 +1,252 @@ +PEP: 497 +Title: Enhanced handling of external requirements +Version: $Revision$ +Last-Modified: $Date$ +Author: Tennessee Leeuwenburg , + +Status: Draft +Type: Standards Track +Python-Version 3.6 +Content-Type: text/x-rst +Created: 03-Aug-2015 +Post-History: 03-Aug-2015 + + +Abstract +======== + +The canonical method for installing packages is to use pip, whether that is +referencing PyPI or a local repository. A limitation of the current install +ecosystem is the relationship with operating-system level dependencies. This +can result in compilation failures or execution failures. Package maintainers +may wish to specify the requirements to avoid this issue, and users may +appreciate earlier stopping and more informative messages about their system. + +The responsibility of pip is to stop early and provide useful information about +missing and not automatically installable dependecies. Ideally it would provide +a useful user message about how to obtain the required dependencies. The +responsibility for actual installation lies with the user at this time. + +This PEP specifies an extension to the requirements sections of setup.py +which will allow authors to express these requirements cleanly. In the +implementation section, a discussion of how this is used to solved problems +is given. + +Here is an example of the proposed syntax, where both operating system and build +target specifiers are included. + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires='yaml, scipy[atlas]', + setup_requires='pyfortran', + external={ + setup=[ + 'bin!fortran:dist_name=='ubuntu', + 'include!kernel.h', + 'include!liblas.h', + ], + install=[ + 'lib!liblas1.so' + ], + } + ) + +It is believe this syntax provides one obvious way to do things which does not +multiply the complexity or parts beyond those necessary to achieve the goal. + +In this example, the user will be presented with a useful install message and +a process that exist early, as opposed to relying on the compiler to fail +and provide an error message that may not be readily understood by most users + +The mechanism of referencing failed dependencies such as 'kernel.h' to useful +user messages regarding the actual operating system packages that are relevant +is discussed in the implementation section, along with the rationale for +handling that referencing outside of setup.py. + +Rationale +========= + +Different environment context have implications for the building, testing, +installing and execution of software. There are two sides to this coin. First, +in some environments, additional python packages are needed. Secondly, +it is useful to specify, e.g., the apt or yum package names which need to be +in place prior to continuing. The syntax proposed here allows for a readable +and consise expression of these requirements. + +Setuptools and pip should be modified to support the use case of installing +the appropriate python packages as required depending on the specification. + +They should also support aborting an install with a useful error message if +required dependecies are not installed in the environment. The responsibility +for actually installing those required packages lies with the user. + +Ideally, and error message will display the package manager command required +to add the missing environment dependencies. + +It is expected that this feature would be commonly used, particularly in +scientific applications which frequently have complex interactions with +externally provided libraries. An example would be needing a version of +numpy and scipy both of which have been compiled to be aware of the ATLAS +compiled set of linear algebra libraries for performance reasons. This sounds +esoteric but is, in fact, a routinely encountered situation which drives +people towards using the anaconda scientific python environment since it +actually solves these issues. + +Syntax +====== + +The valid context specifiers are taken from the environment markers library. +This command says that when the operating system matches 'ubuntu', then add +the yaml==0.3 requirement to the list. + +Implementation +============== + +A common user experience is to attempt the installation of software in a +basic environment, and for the installation to fail during the compilation +of a dependency. The user is presented with an error message provided by the +compiler. Not all users have the ability to understand these messages, and +a common experience is to spend time searching online for the appopriate +package which is provided by the vendor. + +The base implementation will also only provide information about missing +headers, libraries or executables. This is a step forward in that the error +message is provided very early in the install process, rather than potentially +many minutes in to a large compile job. + +A key aspect of the usability of this tool is a mechanism to provide more useful +information to the user about how to obtain those missing resources. +The proposal is to provide a layer inside of pip which is capable of providing +information about appropriate package names. It is surprising that vendors +do not already provide a good tool for this, given that finding appropriate +header and library files is a common experience in the general linux ecosystem. +However, it is also a critical aspect of the user experience, and something +which package maintainers will need to be able to specify precisely. The +proposed approach would be for pip to supply the ability to install an +inference layer, usable by users and vendors alike. Usage would be along +the lines of + + pip install_vendor_magic darwin + +This would transform the user messages relating to missing dependencies from + + Could not find libatlas.so + +to + + Could not find libatlas.so. Run 'port install scipy +atlas' + + +Discussion +========== + +This section is incomplete. Feedback appreciated on relevant discussion +points that should be included. + +Specification of files (headers, libs, bins) vs packages +-------------------------------------------------------- + +There was a discussion about whether specific dependencies (named headers, +libraries, binaries etc) should be included, or whether general package names +(such as may be expected by yum, apt, conda) were appropriate. Due to the +number of package management tools, plus the changing nature of those, it +was decided that concentrating on the specific files was more useful than +using package names. It also provides looser coupling between the specification +of externals and the package management tools. + +Location in setup.py rather than requirements.txt +------------------------------------------------- + +Requirements.txt is scoped to the use for local deployments to specify +requirements precisely for the relevant local environment. Setup.py is the +appropriate place to specify the information for the package in general. + +In future, there may be some scope for including similar directives within +requirements.txt, but that is out of scope for this PEP. + + +Addition of vendor inference layer +---------------------------------- + +Bearing in mind the discussion about using specific dependencies, the ideal +user experience is to get a useful message about how to get the package +installed on their system. This means providing a message which is relevant +for their package management environment (e.g. yum, apt, conda). Each +distributor would be expected to provide an inference lookup which can be +installed into pip which would assist with identifying the packages which +are associated with the named dependencies. + + +Types of environment and vendor +------------------------------- + +Anaconda is not an operation system. Neither is yum, neither is apt. It is the +maintainer of the packaging system which is pertinent, and the current operating +system is an imperfect indicator of the relevant package manager. + + +Directories, categories and locations of dependency files +--------------------------------------------------------- + +The proposed mechanism allows introspection of the standard location for +packages. On *nix environments, these would be (for example), /usr/bin, +/usr/local/include etc (see https://www.gnu.org/prep/standards/html_node/Directory-Variables.html +for a discussion of standard directories). Non-standard dependency locations +would be supported only via the underlying platform's mechanisms for exposing +these details, such as PATH, LD_LIBRARY_PATH etc. This PEP does not include +generalised configuration and specification of dependencies in non-standard +or customised locations. + + + +Alternative Syntax Concept +========================== + +One early straw man for the syntax is included here for reference during +discussions. The chosen approach is based off approaching a small dsl for +handling the specification, which may also point to future directions that +could be taken in requirements.txt. + +:: + + setup( + install_requires='yaml', + setup_requires='pyfortran', + os_requires=[ + ['fortran', type='bin', os='ubuntu', 'target=build'], + ['kernel.h', type='header', os='linux', target='build'], + ['libblas.o', type='lib', target='run'] + ['scipy +atlas', type='packagename', target='run', os='darwin'] + ] + ) + + +References +========== + +.. [1] PEP 426, Metadata for Python Software Packages, Coghlan, Holth, Stufft + (http://www.python.org/dev/peps/pep-0426) + +.. [2] PEP 345, Sample Plaintext PEP Template, Warsaw + (http://www.python.org/dev/peps/pep-0009) + +.. [3] Pull request 29 on interoperability-peps + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From e0f1c0e6b8eaeded373dfe2ea0dcbbb15a6e0c22 Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Mon, 3 Aug 2015 16:24:01 +1000 Subject: [PATCH 2/5] Integrated feedback from Robert Collins --- pep-0497-externals.txt | 75 +++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 19 deletions(-) diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt index 061319e..e441bcd 100644 --- a/pep-0497-externals.txt +++ b/pep-0497-externals.txt @@ -15,20 +15,20 @@ Post-History: 03-Aug-2015 Abstract ======== -The canonical method for installing packages is to use pip, whether that is -referencing PyPI or a local repository. A limitation of the current install -ecosystem is the relationship with operating-system level dependencies. This +The canonical method for installing Python packages is to use pip, whether that +is referencing PyPI or a local repository. A limitation of the current install +ecosystem is the relationship with non-Python dependencies. This can result in compilation failures or execution failures. Package maintainers may wish to specify the requirements to avoid this issue, and users may -appreciate earlier stopping and more informative messages about their system. +appreciate earlier and more informative messages about their system. The responsibility of pip is to stop early and provide useful information about missing and not automatically installable dependecies. Ideally it would provide a useful user message about how to obtain the required dependencies. The responsibility for actual installation lies with the user at this time. -This PEP specifies an extension to the requirements sections of setup.py -which will allow authors to express these requirements cleanly. In the +This PEP proposes a mechanism for extending the requirements section of setup.py +which will allow authors to express non-Python requirements cleanly. In the implementation section, a discussion of how this is used to solved problems is given. @@ -53,12 +53,11 @@ target specifiers are included. } ) -It is believe this syntax provides one obvious way to do things which does not -multiply the complexity or parts beyond those necessary to achieve the goal. -In this example, the user will be presented with a useful install message and -a process that exist early, as opposed to relying on the compiler to fail -and provide an error message that may not be readily understood by most users +If the user does not have the required dependencies, the user will be presented +with a useful install message and a process that exits early, as opposed to +relying on the compiler to fail and provide an error message that may not be +readily understood by most users. The mechanism of referencing failed dependencies such as 'kernel.h' to useful user messages regarding the actual operating system packages that are relevant @@ -68,9 +67,9 @@ handling that referencing outside of setup.py. Rationale ========= -Different environment context have implications for the building, testing, +Differences between environments have implications for the building, testing, installing and execution of software. There are two sides to this coin. First, -in some environments, additional python packages are needed. Secondly, +in some environments, additional Python packages are needed. Secondly, it is useful to specify, e.g., the apt or yum package names which need to be in place prior to continuing. The syntax proposed here allows for a readable and consise expression of these requirements. @@ -184,7 +183,7 @@ Types of environment and vendor Anaconda is not an operation system. Neither is yum, neither is apt. It is the maintainer of the packaging system which is pertinent, and the current operating -system is an imperfect indicator of the relevant package manager. +system is an imperfect indicator of the relevant package manager. Directories, categories and locations of dependency files @@ -201,13 +200,51 @@ or customised locations. -Alternative Syntax Concept +Alternative Syntax Concepts ========================== -One early straw man for the syntax is included here for reference during -discussions. The chosen approach is based off approaching a small dsl for -handling the specification, which may also point to future directions that -could be taken in requirements.txt. +Concept 0: Separated external and non-external dependencies +----------------------------------------------------------- + +As per abstract + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires='yaml, scipy[atlas]', + setup_requires='pyfortran', + external={ + setup=[ + 'bin!fortran:dist_name=='ubuntu', + 'include!kernel.h', + 'include!liblas.h', + ], + install=[ + 'lib!liblas1.so' + ], + } + ) + +Concept 1: Integrate external and non-external dependencies +----------------------------------------------------------- + +:: + + setup( + name='uses_scipy_with_atlas_extension' + install_requires=['yaml', 'scipy[atlas]', 'lib!libblas1.so'], + setup_requires=[ + 'pyfortran', + 'bin!fortran:dist_name=="ubuntu"', + 'include!kernel.h', + 'include!liblas.h', + ], + ) + + +Concept 2: Python-like language +------------------------------- :: From d84a3878f7e8e1b50671915092b031483ca0b6a7 Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Tue, 4 Aug 2015 10:03:36 +1000 Subject: [PATCH 3/5] Responded to feedback from Robert Collins. Expanded and improved upon some sections, edited for clarity --- pep-0497-externals.txt | 108 +++++++++++++++++++++++++++++++---------- 1 file changed, 82 insertions(+), 26 deletions(-) diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt index e441bcd..c28face 100644 --- a/pep-0497-externals.txt +++ b/pep-0497-externals.txt @@ -23,34 +23,30 @@ may wish to specify the requirements to avoid this issue, and users may appreciate earlier and more informative messages about their system. The responsibility of pip is to stop early and provide useful information about -missing and not automatically installable dependecies. Ideally it would provide +missing (and not automatically installable) dependencies. Ideally it would provide a useful user message about how to obtain the required dependencies. The responsibility for actual installation lies with the user at this time. This PEP proposes a mechanism for extending the requirements section of setup.py which will allow authors to express non-Python requirements cleanly. In the -implementation section, a discussion of how this is used to solved problems +implementation section, a discussion of how this may be used to solved problems is given. Here is an example of the proposed syntax, where both operating system and build -target specifiers are included. +target specifiers are included. This is "Concept 1" of the alternatives discussed +later in the document. :: setup( name='uses_scipy_with_atlas_extension' - install_requires='yaml, scipy[atlas]', - setup_requires='pyfortran', - external={ - setup=[ - 'bin!fortran:dist_name=='ubuntu', + install_requires=['yaml', 'scipy[atlas]', 'lib!libblas1.so'], + setup_requires=[ + 'pyfortran', + 'bin!fortran:dist_name=="ubuntu"', 'include!kernel.h', 'include!liblas.h', ], - install=[ - 'lib!liblas1.so' - ], - } ) @@ -69,10 +65,10 @@ Rationale Differences between environments have implications for the building, testing, installing and execution of software. There are two sides to this coin. First, -in some environments, additional Python packages are needed. Secondly, -it is useful to specify, e.g., the apt or yum package names which need to be -in place prior to continuing. The syntax proposed here allows for a readable -and consise expression of these requirements. +in some environments, additional Python packages are needed. Secondly, in some +environments, additional non-Python packages need to be present, such as header +files, libraries, scripts or other binaries. The syntax proposed here allows +the description of both sorts of requirements in a single syntax. Setuptools and pip should be modified to support the use case of installing the appropriate python packages as required depending on the specification. @@ -81,17 +77,21 @@ They should also support aborting an install with a useful error message if required dependecies are not installed in the environment. The responsibility for actually installing those required packages lies with the user. +This would allow automatic installation of environment-specific Python +dependencies and pave the way to improving how users go about installing +non-Python prerequisites. + Ideally, and error message will display the package manager command required to add the missing environment dependencies. It is expected that this feature would be commonly used, particularly in scientific applications which frequently have complex interactions with -externally provided libraries. An example would be needing a version of -numpy and scipy both of which have been compiled to be aware of the ATLAS +externally provided libraries. An example would be needing versions of +numpy and scipy, both of which must have been compiled to be aware of the ATLAS compiled set of linear algebra libraries for performance reasons. This sounds esoteric but is, in fact, a routinely encountered situation which drives -people towards using the anaconda scientific python environment since it -actually solves these issues. +people towards using the alternative packaging for scientific python +environments. Syntax ====== @@ -100,8 +100,45 @@ The valid context specifiers are taken from the environment markers library. This command says that when the operating system matches 'ubuntu', then add the yaml==0.3 requirement to the list. -Implementation -============== +Meta: I am not a syntax wizard and will need help from others to properly specify +the syntax as a formal grammar if that is required. + +Essentially, the "!" separator is used to flag key-value pairs which will allow +two things: the specification of dependency type and the environment context +where the rule should apply. + +For example, consider the line + + setup_requires=['bin!fortran:dist_name=="ubuntu"'] + +The 'bin' specifies one of "bin", "lib" or "header", representing the only +non-Python dependency types allows. The absence of this prefix implies the the +package name is a discoverable Python package. + +Following the ':' is a comma-separated list of key-operator-value pairs. These +represent the sufficient trigger conditions to apply the rule. At this stage +only a very limited set of allowable operators and logics are permitted. The +goal is to provide just enough support to allow the specification of key +dependencies for specific environments, not to become a make/build system. + +Implementation Concerns +======================= + +Short Version +------------- + +1. Setuptool + pip should display relevant information earlier than now +2. A mechanism to assist in identifying external package names would be useful +3. The maintenance burden for (2) should fall on the package management system + maintainers, not on pip/setuptools developers +4. This means a simple pluging approach to mapping from specific resources + to relevant external package names and invocation calls +5. I propose this looks like "pip install vendor_magic darwin" +6. And that the various O/Ses ship this pre-installed in their distributions + + +Discussion +---------- A common user experience is to attempt the installation of software in a basic environment, and for the installation to fail during the compilation @@ -127,7 +164,7 @@ proposed approach would be for pip to supply the ability to install an inference layer, usable by users and vendors alike. Usage would be along the lines of - pip install_vendor_magic darwin + pip install vendor_magic darwin This would transform the user messages relating to missing dependencies from @@ -141,8 +178,8 @@ to Discussion ========== -This section is incomplete. Feedback appreciated on relevant discussion -points that should be included. +This section will be updated to capture incoming feedback and represents a +snapshot of the current concerns. Specification of files (headers, libs, bins) vs packages -------------------------------------------------------- @@ -206,7 +243,14 @@ Alternative Syntax Concepts Concept 0: Separated external and non-external dependencies ----------------------------------------------------------- -As per abstract +This is an early-stage syntax. Concept 1 replaces it as the current +"front runner" as implementation candidate and is included in the abstract. + +The advantage of Concept 0 is that is clearly separates the non-Python +dependency specification into an isolated part of the configuration arguments, +reducing the potential for confusion between external and internal dependencies. +This may assist during implementation as no 'sniffer' will be needed to +determine whether a named package is internal or external. :: @@ -229,6 +273,12 @@ As per abstract Concept 1: Integrate external and non-external dependencies ----------------------------------------------------------- +A good argument for integrating these is that it will allow extensions +to requirements.txt without needing a separate file to exist for the externals. +It is shorter and still highly readable. Potentially instead of e.g. "lib!" +for a prefix, "extlib!" could be used to signpost more strongly. But it's +longer. + :: setup( @@ -246,6 +296,12 @@ Concept 1: Integrate external and non-external dependencies Concept 2: Python-like language ------------------------------- +This isn't really valid code and is included mainly to document the development +of ideas. It illustrates an alternative structure to using separate config +areas for setup, install etc and treats those as a part of the trigger +for a particular dependency, called the target. There is still some advantage +to the clarity and flexibility provided by this approach. + :: setup( From 3c06cd8f8785aa4bc39eadbe1431f90a6c1bd95a Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Tue, 4 Aug 2015 10:44:01 +1000 Subject: [PATCH 4/5] Added reference to prior PEPs based on PKG-INFO and modified title based on feedback --- pep-0497-externals.txt | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt index c28face..fb6f1af 100644 --- a/pep-0497-externals.txt +++ b/pep-0497-externals.txt @@ -1,5 +1,5 @@ PEP: 497 -Title: Enhanced handling of external requirements +Title: External and environment-specific dependencies Version: $Revision$ Last-Modified: $Date$ Author: Tennessee Leeuwenburg , @@ -137,8 +137,8 @@ Short Version 6. And that the various O/Ses ship this pre-installed in their distributions -Discussion ----------- +Discussion of Implementation Concerns +------------------------------------- A common user experience is to attempt the installation of software in a basic environment, and for the installation to fail during the compilation @@ -175,12 +175,13 @@ to Could not find libatlas.so. Run 'port install scipy +atlas' -Discussion -========== +Discussion / FAQ / Issues +========================= This section will be updated to capture incoming feedback and represents a snapshot of the current concerns. + Specification of files (headers, libs, bins) vs packages -------------------------------------------------------- @@ -202,6 +203,19 @@ appropriate place to specify the information for the package in general. In future, there may be some scope for including similar directives within requirements.txt, but that is out of scope for this PEP. +Prior Approaches +---------------- + +Some capacity to support description of external dependencies (among other +things) was described in PEPs 314 and 345 (Metadata 1.1 and 1.2). These +describe directives whichs can be placed in the PKG-INFO file. An implementation +responding to these directives is pending. + +This PEP places the directives in setup.py. This approach is closer to the +typical approach of package maintainers to expressing requirements and places +the external dependency specification nearer to the Python dependency +specification. This PEP would supercede PEPs 314 and 345 with respect to +external dependencies. Addition of vendor inference layer ---------------------------------- From 9d91e47dc4070decc11c2d2521ca891fa6093638 Mon Sep 17 00:00:00 2001 From: Tennessee Leeuwenburg Date: Tue, 4 Aug 2015 15:22:43 +1000 Subject: [PATCH 5/5] Responded to many of the comments raised by Robert and Nick --- pep-0497-externals.txt | 199 +++++++++++++++++++++++++++++++---------- 1 file changed, 150 insertions(+), 49 deletions(-) diff --git a/pep-0497-externals.txt b/pep-0497-externals.txt index fb6f1af..223683f 100644 --- a/pep-0497-externals.txt +++ b/pep-0497-externals.txt @@ -1,5 +1,5 @@ PEP: 497 -Title: External and environment-specific dependencies +Title: External dependencies Version: $Revision$ Last-Modified: $Date$ Author: Tennessee Leeuwenburg , @@ -11,30 +11,47 @@ Content-Type: text/x-rst Created: 03-Aug-2015 Post-History: 03-Aug-2015 +Very Short Abstract +=================== + +This PEP covers three things. + +1. How users can specify external dependencies in setup.py +2. How users can specify external dependencies in requirements.txt +3. How external dependencies will be communicated between tools, + (setuptools --> wheel, wheel --> pip, ...) + +It recommends additional syntax to specify the requirements, implementation +recommendations on how to provide appropriate messages to users, and relevant +detailed discussion on technical issues. Where only setup.py is referenced, +the scope should be taken to include requirements.txt unless otherwise +mentioned. Abstract ======== -The canonical method for installing Python packages is to use pip, whether that -is referencing PyPI or a local repository. A limitation of the current install -ecosystem is the relationship with non-Python dependencies. This -can result in compilation failures or execution failures. Package maintainers -may wish to specify the requirements to avoid this issue, and users may -appreciate earlier and more informative messages about their system. +The canonical method for installing Python packages is to use pip and/or +setuptools, whether that is referencing PyPI or a local repository. A limitation +of the current install ecosystem is the relationship with non-Python +dependencies. This can result in compilation failures or execution failures. +Package maintainers may wish to specify the requirements to avoid this issue, +and users may appreciate earlier and more informative messages about their +system. The responsibility of pip is to stop early and provide useful information about missing (and not automatically installable) dependencies. Ideally it would provide a useful user message about how to obtain the required dependencies. The responsibility for actual installation lies with the user at this time. -This PEP proposes a mechanism for extending the requirements section of setup.py -which will allow authors to express non-Python requirements cleanly. In the -implementation section, a discussion of how this may be used to solved problems -is given. +This PEP proposes a mechanism for extending the requirements section of +setup.py, and the content of requirements.txt, which will allow authors to +express non-Python requirements cleanly. In the implementation section, a +discussion of how this may be used to solve problems is given. -Here is an example of the proposed syntax, where both operating system and build -target specifiers are included. This is "Concept 1" of the alternatives discussed -later in the document. +Here is an example of the proposed syntax for setup.py, where both operating +system and build target specifiers are included. This is "Concept 1" of the +alternatives discussed later in the document. The syntax for requirements.txt +is similar, and is included later in the document. :: @@ -49,7 +66,6 @@ later in the document. ], ) - If the user does not have the required dependencies, the user will be presented with a useful install message and a process that exits early, as opposed to relying on the compiler to fail and provide an error message that may not be @@ -81,31 +97,33 @@ This would allow automatic installation of environment-specific Python dependencies and pave the way to improving how users go about installing non-Python prerequisites. -Ideally, and error message will display the package manager command required +Ideally, any error messages will include the package manager command required to add the missing environment dependencies. It is expected that this feature would be commonly used, particularly in scientific applications which frequently have complex interactions with -externally provided libraries. An example would be needing versions of -numpy and scipy, both of which must have been compiled to be aware of the ATLAS -compiled set of linear algebra libraries for performance reasons. This sounds -esoteric but is, in fact, a routinely encountered situation which drives -people towards using the alternative packaging for scientific python -environments. +externally provided libraries. The example provided in the abstract is a +hypothetical package which needs versions of numpy and scipy, both of which +must have been compiled to be aware of the ATLAS compiled set of linear algebra +libraries (for performance reasons). This sounds esoteric but is, in fact, a +routinely encountered situation which drives people towards using the +alternative packaging for scientific python environments. Syntax ====== -The valid context specifiers are taken from the environment markers library. -This command says that when the operating system matches 'ubuntu', then add -the yaml==0.3 requirement to the list. +The proposed syntax is an external object type, an "!", and then an external +object name. The object type can be one of: + + include + bin + lib -Meta: I am not a syntax wizard and will need help from others to properly specify -the syntax as a formal grammar if that is required. +The object name should be the name of the required thing - search paths are +presumed to be used to find the object. -Essentially, the "!" separator is used to flag key-value pairs which will allow -two things: the specification of dependency type and the environment context -where the rule should apply. +These dependencies may be controlled via environment markers in the standard +fashion (see PEP 496). For example, consider the line @@ -113,13 +131,8 @@ For example, consider the line The 'bin' specifies one of "bin", "lib" or "header", representing the only non-Python dependency types allows. The absence of this prefix implies the the -package name is a discoverable Python package. - -Following the ':' is a comma-separated list of key-operator-value pairs. These -represent the sufficient trigger conditions to apply the rule. At this stage -only a very limited set of allowable operators and logics are permitted. The -goal is to provide just enough support to allow the specification of key -dependencies for specific environments, not to become a make/build system. +package name is a discoverable Python package. This rule is controlled via the +environment marker syntax to apply only on the Ubuntu distribution. Implementation Concerns ======================= @@ -136,6 +149,64 @@ Short Version 5. I propose this looks like "pip install vendor_magic darwin" 6. And that the various O/Ses ship this pre-installed in their distributions +Platform Abstraction +-------------------- + +Pip and setuptools cannot (and should not) be expected to provide detailed +reasoning about the external environment and appropriate responses. Questions +have arisen often about the degree to which the Python ecosystem build tools +can or should auto-detect or otherwise handle the collection of external +dependencies at build time. + +Extending the Python build and packaging system into the external environment +space multiplies the responsibility of package authors excessively. The entities +which are appropriately informed and resourced to handle the complexities +of complex build needs are the environment tool maintainers. + +The issue that is being solved is a reasonable communications layer by which +information can be shared between those two separated ecosystems. Python +package maintainers who are aware they are building a system-dependent package +can take steps to represent known critical dependencies in their configurations +so that developers can discover these more easily when building or installing +tools. This is, in part, due to the fact that standard build tools often fail +late in the build process and themselves provide less information that is ideal +to end users. + +Given the broad use of Python by many users, providing additional information +here is extremely useful. Further, these tools provide the means for external +package systems to respond appropriately to the dependency needs of Python +packages. + + +Setuptools +---------- + +Wheel +----- + +Pip +--- + +Case sensitivity of specifiers +------------------------------ + +Hooks, stages and processes +--------------------------- + +1. Perform checks of availability of resources +2. Provide a callback hook to external tools. These hooks can return either + failure (no recourse) or success (resource provided). In this way, it would + be possible to support logging (returns failure but has a useful side-effect) + or a problem-resolution response (actually installs the external dependency) +3. If there are remaining unresolved dependencies, proceed to an early exit +4. Provide a callback hook indicating early exit along with appropriate + messaging indicating the unresolved external dependencies +5. Provide final messages to the user + +The vendor magic layer would be implemented as a stage four callback tool which +was capable of understanding the resource specifications and messaging, and +supplement the standard pip/setuptools messaging with appropriate information +for appropriate packages to install and/or suitable actions the user could take. Discussion of Implementation Concerns ------------------------------------- @@ -154,15 +225,15 @@ many minutes in to a large compile job. A key aspect of the usability of this tool is a mechanism to provide more useful information to the user about how to obtain those missing resources. -The proposal is to provide a layer inside of pip which is capable of providing -information about appropriate package names. It is surprising that vendors -do not already provide a good tool for this, given that finding appropriate -header and library files is a common experience in the general linux ecosystem. -However, it is also a critical aspect of the user experience, and something -which package maintainers will need to be able to specify precisely. The -proposed approach would be for pip to supply the ability to install an -inference layer, usable by users and vendors alike. Usage would be along -the lines of +The proposal is to provide a layer, consumed by pip, which is capable of +providing information about appropriate package names. It is surprising that +vendors do not already provide a good tool for this, given that finding +appropriate header and library files is a common experience in the general +linux ecosystem. However, it is also a critical aspect of the user experience, +and something which package maintainers will need to be able to specify +precisely. The proposed approach would be for pip to supply the ability to +install an inference layer, usable by users and vendors alike. Usage would be +along the lines of pip install vendor_magic darwin @@ -170,9 +241,9 @@ This would transform the user messages relating to missing dependencies from Could not find libatlas.so -to +to, e.g., - Could not find libatlas.so. Run 'port install scipy +atlas' + Could not find libatlas.so. Run 'port install atlas' Discussion / FAQ / Issues @@ -228,11 +299,21 @@ distributor would be expected to provide an inference lookup which can be installed into pip which would assist with identifying the packages which are associated with the named dependencies. +Implementation of the vendor inference layer +-------------------------------------------- + +There may be some advantage to regard the vendor inference layer as TBD in +a future PEP due to the complexity and also in order to better focus the scope +of the PEP. This externals PEP can proceed effectively without the vendor +inference layer being implemented or specified. Alternatively, a separate PEP +could be developed in tandem with this proposal to hold the apppropriate level +of detail. + Types of environment and vendor ------------------------------- -Anaconda is not an operation system. Neither is yum, neither is apt. It is the +Anaconda is not an operating system. Neither is yum, neither is apt. It is the maintainer of the packaging system which is pertinent, and the current operating system is an imperfect indicator of the relevant package manager. @@ -249,7 +330,20 @@ these details, such as PATH, LD_LIBRARY_PATH etc. This PEP does not include generalised configuration and specification of dependencies in non-standard or customised locations. +The Python installation tools should honour those standard mechanisms. +The details are per-platform and not specified by this specification +(see Platform Abstraction for more on this point). + +Virtual Environment Concerns +---------------------------- +This should work fine with virtual environments. Since the PEP draws the line +at performing external package installation, it is up to package management +tools to decide how to separate concerns between systemwide installation +and virtual environment installation. For example, a system such as +Anaconda which is heavily based around virtual environments is well capable +of handling the installation of external dependencies into a virtual environment +context. Alternative Syntax Concepts ========================== @@ -293,6 +387,13 @@ It is shorter and still highly readable. Potentially instead of e.g. "lib!" for a prefix, "extlib!" could be used to signpost more strongly. But it's longer. +Another advantage to this approach it that any tool which treats setup_requires +(et al) as a list of opaque strings should "just work", while a separate field +causes problems. + +Notably, this change should mostly be forwards compatible with the overall +structure of PEP 426. + :: setup(