Option for case insensitive search at runtime

[markharwood]

This meta issue tracks the various changes relating to offering a "case insensitive" option to various term-level queries (term, terms, prefix, wildcard, regex) at search time. It replaces the previous #53603 which meandered with various discussions.
In query DSL we will offer a new case_insensitive flag which can only be set to true to enable new behaviour. When left unset the existing behaviour is used (which is inconsistent - keyword fields with normalizers normalize query terms while text fields do not). Due to these inconsistencies and lack of guarantees, setting the case_insensitive flag to false will throw an error.

Tasks

• Add case insensitive option to Lucene's RegExpQuery - merged RegExp - add case insensitive matching option apache/lucene-solr#1541
• regexp queries in elasticsearch DSL - Issue PR Search - add case insensitive support for regex queries. #59441
• Lucene query_string support for /Foo/i regex syntax - PR LUCENE-9445 Add support for case insensitive regex searches in QueryParser apache/lucene-solr#1708
• term queries in elasticsearch DSL- issue Search - add case insensitive flag to term, prefix and wildcard queries #61546
• terms queries in elasticsearch DSL
• prefix queries in elasticsearch DSL - issue Search - add case insensitive flag to term, prefix and wildcard queries #61546
• wildcard query in elasticsearch DSL - issue Search - add case insensitive flag to term, prefix and wildcard queries #61546

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[markharwood]

@jimczi / @jpountz - shall I open Lucene issues to add case insensitive param to TermQuery, PrefixQuery and WildcardQuery or create case insensitive variants for elasticsearch?

[jpountz]

I wonder if we really need new queries or if we could reuse AutomatonQuery to build case-insensitive variants?

[markharwood]

I wonder if we really need new queries or if we could reuse AutomatonQuery to build case-insensitive variants?

PrefixQuery and WildcardQuery already do reuse AutomatonQuery? They're lightweight subclasses that take args and implement a toAutomaton() function. The case insensitive options we want to add can either be added as an "if" statement in core Lucene or we fork those classes as something like this:

public class CaseInsensitivePrefixQuery extends CaseInsensitiveAutomatonQuery {

/** Constructs a case insensitive query for terms starting with <code>prefix</code>. */
public CaseInsensitivePrefixQuery(Term prefix) {
    super(prefix, toAutomaton(prefix.bytes()), Integer.MAX_VALUE, true);
}

/** Build an automaton accepting all terms with the specified prefix, case insensitive. */
public static Automaton toAutomaton(BytesRef prefix) {
    if (prefix == null) {
        throw new NullPointerException("prefix must not be null");
    }
    List<Automaton> list = new ArrayList<>();
    String s = prefix.utf8ToString();
    Iterator<Integer> iter = s.codePoints().iterator();
    while (iter.hasNext()) {
        list.add(toCaseInsensitiveChar(iter.next(), Integer.MAX_VALUE));
    }
    list.add(Automata.makeAnyString());

    Automaton a = Operations.concatenate(list);
    a = MinimizationOperations.minimize(a, Integer.MAX_VALUE);
    return a;
}

The CaseInsensitiveAutomatonQuery base class proposed above offers toCaseInsensitiveChar helper function to help create [Ff][Oo][Oo] type sequences from foo input