span_not's documentation is misleading and "post" may not work as expected

[gcampbell-epiq]

Elasticsearch version documentation 5.6, runtime 5.3.2
Plugins installed: none
JVM version 1.8.0_71
OS version Windows Server 2008

span_not Documentation Issue

Description of the problem including expected versus actual behavior:

Documentation on all span queries is very sparse, particularly for the span_not. Deviating slightly from the example produces unexpected results.

Steps to reproduce:

1. Create default index
   PUT span-not-test
2. Insert docs containing "la hoya" and "la hoya hoya"

PUT span-not-test/doc/1
{
    "field1": "la hoya"
}
PUT span-not-test/doc/2
{
    "field1": "hoya la hoya"
}
PUT span-not-test/doc/3
{
    "field1": "la hoya hoya"
}

3. Run example search modified slightly to increase slop and flip in_order to false

POST span-not-test/_search
{
    "query": {
        "span_not" : {
            "include" : {
                "span_term" : { "field1" : "hoya" } 
            },
            "exclude" : {
                "span_near" : {
                    "clauses" : [
                        { "span_term" : { "field1" : "la" } },
                        { "span_term" : { "field1" : "hoya" } }
                    ],
                    "slop" : 50,
                    "in_order" : false
                }
            }
        }
    }
}

Expected results:
No docs should be returned, because in the all test docs contain "la" and "hoya" within 100 tokens of eachother.

Actual results:
Doc 3 is returned.

I assume the issue is that the "included" runs of tokens are not checked for "exclusion" if they overlap with another run of included tokens.

Whatever the case, I understand that using dist/pre/post would yield correct results, but the documentation does not help me understand why this is the case.

SpanNotQuery post Issue

I have some single-value fields, the length of which is unlimited. In order to avoid secretly imposing a length limit and always getting valid results from a span_not query, I attempted to use int.max (2^31) as the dist in a span_not query. The issue is that SpanNotQuery took that dist value and added to it the end position of an "included" token. That yielding ~-2B. That negative position was then compared against an "excluded" term's position to determine if the excluded term's position was less, i.e. within dist of the included term. So, even though my dist was high, my excluded term was not detected in a position within range of my included term.

You can see the relevant code here in the Lucene repo.
~\core\src\java\org\apache\lucene\search\spans\SpanNotQuery.java
line 181

[jimczi]

No docs should be returned, because in the all test docs contain "la" and "hoya" within 100 tokens of eachother.

The span_not query removes matches which overlap with another SpanQuery or which are within x tokens before or y tokens after another SpanQuery.
In the example query, documents that contain hoya alone are considered as a match. This is the expected behavior, hoya la hoya will match because hoya is within 1 token of la hoya.
As you noticed pre and post can be used to control the x and y which default to 0.
I opened #27150 to clarify this behavior in the documentation.

If you want to remove span_near matches exclusively you can use a simple bool query:

{  
   "query":{  
      "bool":{  
         "must":[  
            {  
               "span_term":{  
                  "field1":"hoya"
               }
            }
         ],
         "must_not":[  
            {  
               "span_near":{  
                  "clauses":[  
                     {  
                        "span_term":{  
                           "field1":"la"
                        }
                     },
                     {  
                        "span_term":{  
                           "field1":"hoya"
                        }
                     }
                  ],
                  "slop":0,
                  "in_order":true
               }
            }
         ]
      }
   }
}

The span_not query implies a relation between the include and exclude clause which is why it's an advanced case that requires some understanding of span queries in general.

[gcampbell-epiq]

Thanks, Jim. I'll look into filing a bug with Lucene on the other issue.

[gcampbell-epiq]

Closed by accident, sorry.

[jimczi]

I'll look into filing a bug with Lucene on the other issue.

It's not really a bug, just an overflow if you use a very high dist value. Don't do that and use a must_not clause instead. dist (and the others) could be limited to some values on the es side but that's really a detail if you know what this option does.