Expose new case insensitive RegExpQuery support in QueryParser [LUCENE-9445]

[asfimport]

#10426 added a case insensitive matching option to RegExpQuery.

This proposal is to extend the QueryParser syntax to allow for an optional i (case Insensitive) flag to appear on the end of regular expressions e.g. /Foo/i

 

This is regex syntax supported by a number of programming languages.

---

Migrated from LUCENE-9445 by Mark Harwood (@markharwood), updated Sep 17 2020
Linked issues:

• Clean up obsolete and commented-out cruft from StandardSyntaxParser.jj [LUCENE-9528] #10568
• Update javacc to the latest version [LUCENE-9526] #10566

Pull requests: apache/lucene-solr#1708

[asfimport]

Mark Harwood (@markharwood) (migrated from JIRA)

We will likely have to break existing parser behaviour to add this feature.

The existing parser impl allows regex clauses and other clauses to appear next to each other without a space e.g.

  /foo/bar

is interpreted as a regex query "foo" OR term query "bar".

If we want to introduce /Foo/i as syntax for a case insensitive regex then we will need to insist on a space between regexes and other search terms to cleanly separate any regex flags from other search terms. This will require throwing an error if we encounter any text other than "i" after the closing /. It will still be legal to have operators like ) for boolean logic or ^ for boosts immediately after the regex closing slash but any other tokens will cause an error. I have discussed with Jim Ferenczi the idea of a flag to control what happens in an error situation but we can't easily revert to parsing the offending string in a BWC way and it's not ideal to silently drop it either. Always throwing an error seems like the only viable option.

The repercussions of stricter parsing is that sloppy searches like a cut-and-paste URL or file path would now throw an error. These would fail loudly for example:

  [http://foo.com/bar]
  /mydrive/myfolder/myfile

 

The question is does this new regex syntax warrant the breaking change this introduces, or more broadly, what is the BWC policy for making changes to query parser?

 

 

 

 

[asfimport]

Dawid Weiss (@dweiss) (migrated from JIRA)

I've just spent a day cleaning up the grammar for flexible query parser, Mark... Do you think you can wait until I push it? #10568

[asfimport]

Mark Harwood (@markharwood) (migrated from JIRA)

Sure, Dawid.

I'll need to make some changes to throw errors anyway if we agree stricter parsing is the way forward.

[asfimport]

Dawid Weiss (@dweiss) (migrated from JIRA)

Thanks Mark. I think most query parsers offering "advanced" features should be stricter - they should throw errors when they detect a syntax problem. See my e-mail to the mailing list about the current behavior of "/foo/~2"...

[asfimport]

Mark Harwood (@markharwood) (migrated from JIRA)

Yikes. There look to be a number of silent failures currently and some extra strictness is welcome