SPARQL engine breaks on import with next major pyparsing library

[ajnelson-nist]

Hi all,

I found a looming incompatibility with a dependency that is currently in alpha status for its next major release. If this compatibility issue is not addressed, the SPARQL engine in rdflib will not import.

rdflib currently imports pyparsing at version 2.4.7. In a few inconsistently-reproducing cases, I'd found that adding this line to a Python script would cause an import failure: import rdflib.plugins.sparql. Stack trace:

source venv_broken/bin/activate \
  && python3 -c 'import pyparsing ; print(pyparsing.__version__)'
3.0.0a2
source venv_broken/bin/activate \
  && python3 -c 'import rdflib.plugins.sparql'
RDFLib Version: 5.0.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ajn/local/src/test/venv_broken/lib/python3.6/site-packages/rdflib/plugins/sparql/__init__.py", line 33, in <module>
    from . import parser
  File "/home/ajn/local/src/test/venv_broken/lib/python3.6/site-packages/rdflib/plugins/sparql/parser.py", line 184, in <module>
    Param('prefix', PN_PREFIX)) + Suppress(':').leaveWhitespace()
  File "/home/ajn/local/src/test/venv_broken/lib/python3.6/site-packages/rdflib/plugins/sparql/parserutils.py", line 114, in __init__
    self.name = name
AttributeError: can't set attribute

Here is that line in today's master branch for context.

Discovery background:

I'm still not sure how I've been triggering pip to download an alpha release. My best guess at the moment comes from this only happening when I run python setup.py install in a project's source tree to install the project into a virtual environment for some tests. A bit maddeningly, this has happened in my Mac environment running Python 3.8 from MacPorts; an Ubuntu 18.04 environment running Python 3.6; but not an Ubuntu 20.04 environment running Python 3.8.

Minimal steps to reproduce:

1. Create virtual environment.
2. Install packages with this requirements file:

pyparsing == 3.0.0a2
rdflib
requests

3. Activate environment, and in some script import rdflib.plugins.sparql.

EDITED TO ADD: I'd accidentally truncated the last line of the stack trace reporting the AttributeError.

[cmungall]

Not sure if this helps, but another way to reproduce is to use Pipfile with allow_prereleases = true

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
rdflib = "~=5.0"
requests = "~=2.26"

[dev-packages]
tox = "*"

[pipenv]
allow_prereleases = true

This has the effect of installing 3.0.0b2 of pyparsing

Perhaps this is not surprising, but I am inexperienced with pipenv and had inherited the config, and needed allow_prereleases = true for other packages.

If anyone stumbles across the same problem, if you want to keep using allow_prereleases = true then you will need to pin pyparsing to the 2.x series, e.g. by pyparsing = "~=2.4" to be safe

But surely the best solution would be to pin
https://github.com/RDFLib/rdflib/blob/master/requirements.txt
to == 2.*?

Otherwise I assume when pyparsing 3 is released a lot of code will break

[ajnelson-nist]

In my opinion (mine only, not my employer's, etc.), the correct answer is to put a version-check in the code to see if the major version of pyparsing is 3 or not. The current logic can be kept for version < 3, but new logic should be written for version >= 3.

I don't think pinning the major version of pyparsing is healthy in the end, because it causes a dependent-package constraint that could become difficult to satisfy down the road, and holds rdflib projects back from updates to the pyparsing package. So, respectfully, I oppose PR 1366.

I think somebody still needs to take some time and prepare a fix that will work when pyparsing's release happens.

[cmungall]

I think your approach is much better long term, but is the patch to pin versions necessarily wrong as an interim measure?

I think projects are already hindered from updating pyparsing to v3 beta, as this will cause runtime errors.

I think if someone can go ahead and implement the logic that would be fantastic, but this has been open for 9 months and no one seems to have come forward with a fix.

[nicholascar]

I agree that the proposal to actually fix things is better but I'm merging this PR now so we can guarantee stability.

We have reduced the number of long-term open PRs a lot recently but we are a long way from addressing many of the hundreds of open issues. Many are likely resolved but not closed.

@ajnelson-nist please submit a follow-up Issue to keep this problem live. A new Issue to mention it would be better than harking back to an ancient Issue.