EQL: Remove pattern `== "wild*card"`

[costin]

Currently EQL supports the following pattern for doing wildcard comparisons:
field == "wild*card" which translates to wildcard(field, "wild*card").
That is making a comparison (== or !=) to a string that contains * translates to a wildcard check.

This is convenient however it has a few side-effects:

1. one cannot make a comparison to a string containing *.
2. the comparison/wildcard inherits the semantics of equals such as case-insensitivity.
3. makes it hard to reason whether an expression is string equality or wildcard comparison:
   foo == concat("wild", "*", "card")
4. for more than one pattern, the pattern ends up being more verbose:
   foo == "wild*" or foo == "*card" vs wildcard(foo, "wild*", "*card")

It would be good to get some numbers to see how often this pattern is being used.
My proposal is to deprecate/remove this syntactic sugar and promote explicit use of wildcard for clear semantics and clarity in intent.

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[rw-access]

It would be good to get some numbers to see how often this pattern is being used.

If I were to take a SWAG, I would say approximately 95%+ of queries use this behavior currently.

[rw-access]

I got some more stats

• 84% of our production rules use wildcards, averaging 2.5 per rule. There are actually many more, but under the hood in the EQL engine for python, adjacent usages with an or get compressed into one. I don't have numbers for Endgame users or other EQL in the wild.

The ease of wildcards was intentionally baked into EQL from day one, so I think we should approach it very carefully before we make plans to change this behavior. And it could be good to share some context to better understand the expectations and desires of EQL users and rule writers.

When writing detections, you often don't have the full text and know exactly what you're looking for, so wildcards are important. Wildcards are so prevalent that we deliberately made them first class, letting == and != work for wildcard comparisons.

I think it shows success that we've made it this long without needing to express a literal asterisk in a string comparison. All of these concerns have been hypothetical, and that really goes to show the value of and asterisk as a wildcard character. And we aren't the only database/query language to recognize how its importance outweighs the utility of a literal asterisk.

Addressing some of the side-affects you mentioned:

1. one cannot make a comparison to a string containing *.

There are workarounds to do this, but I will concede that they aren't ideal. One example: some_field in ("string with literal * asterisk")

2. the comparison/wildcard inherits the semantics of equals such as case-insensitivity.

Agreed, but I'm not sure how this is a unique side-effect

3. makes it hard to reason whether an expression is string equality or wildcard comparison

String equality is semantically identical to a wildcard comparison but without any wildcard characters, so I don't see an issue here. If the example you provided, foo == concat("wild", "*", "card") is converted to a wildcard under the hood, then I would argue that it's a bug. You could make an argument that it's Undefined Behavior for EQL, but there is no documentation to suggest that this would be a wildcard comparison. All of the examples using wildcards use literal strings. Neither the original documentation and the elasticsearch documentation makes no mention of wildcards for folded values.

4. for more than one pattern, the pattern ends up being more verbose
   foo == "wild*" or foo == "*card" vs wildcard(foo, "wild*", "*card")

Yes, but this sounds like a feature not a bug.

Outstanding concerns
The biggest problem I have is that this is the worst kind of breaking change: a quiet one.

My proposal is to deprecate/remove this syntactic sugar and promote explicit use of wildcard for clear semantics and clarity in intent.

It's impossible to deprecate this syntactic sugar, without coming up with a new character for quotes. Not to be a broken record, but we can't quietly (aka no messages) change the meaning of existing syntax. Otherwise, users will have zero idea why queries that used to work no longer work.

TL;DR
If I had to sum up this approach in a quick sentence it would be this:
"It's a feature, not a bug."

[astefan]

I agree that this is an important feature @rw-access

The ease of wildcards was intentionally baked into EQL from day one, so I think we should approach it very carefully before we make plans to change this behavior. And it could be good to share some context to better understand the expectations and desires of EQL users and rule writers.

When writing detections, you often don't have the full text and know exactly what you're looking for, so wildcards are important. Wildcards are so prevalent that we deliberately made them first class, letting == and != work for wildcard comparisons.

but I cannot go over the inconsistency it adds to the language. The examples @costin provided show that a single use of the language could translate into two separate behaviors that cannot be reasoned about in a deterministic way. Is the user intention when writing foo == concat("wild", "*", "card") either to be an exact match or a wildcard match? It could be that the user really wants to search for wild*card (exact match), even though the vast majority of other uses aim for the second way of using wildcards and one could argue that we make an exception in this case. But still, the language needs to be consistent and deterministic. Also, just like the other features that we want to evaluate at the eql language, proposed changes could be deprecated in other eql implementations following those products deprecation flow and procedures, but for ES EQL and its experimental current status, these could go in without any notice or warnings.

[matriv]

This is a difficult decision, because unlike replacing backquote with single quote were we have a clear behaviour and an error message, if we make this change users could come up with wrong results. On the other hand, I'm not fan of such syntactic sugar because although it can ease the life of a user in the common case, it can have ambiguous semantics as for example in the case of foo == concat('wild', '*', 'card'). Is it a way to bypass the translation to wildcard() function or is still considered a wildcard? What about foo == ?"wild*card"? What's happening for sequence matching keys? if we have an event/doc with value for key= "wild*card", and another event/doc with key="wildXXXXcard" should they match?

Personally I'd always prefer clear semantics to user convenience. Imho. the == operator is the strongest operation for a filter in a query language, and it shouldn't have any syntactic sugar attached that translates it to something else. If we had to design the language from scratch, I'd rather have an operator like ~= as syntactic sugar, which translates the * to a wildcard query, than changing the semantics of EQUAL based on the the string literal.

Taking into consideration the fact that the change of this behaviour is "silent" and can lead to unexpected results, If we decide to make the change, and I'd vote towards it, let's make it now in the experimental stage, otherwise it will be impossible.

[costin]

tl;dr

== is arguably the most important operator in bool clauses. Its semantics and properties are well understood yet EQL breaks them.
I'd like to get agreement that his is a bug and needs changing. If not, how do we resolve the inconsistencies brought by it.

I don't want to get side-tracked by the actual details and timeline (deprecation, compatibility flags, etc...).
That is a separate discussion especially considering the high usage of ==.

To your point on:

If the example you provided, foo == concat("wild", "*", "card") is converted to a wildcard under the hood, then I would argue that it's a bug. You could make an argument that it's Undefined Behavior for EQL, but there is no documentation to suggest that this would be a wildcard comparison.

// string eq
"abc" == concat("ab", "c")  // TRUE 
"abc" == concat("a", "bc")  // TRUE
// match eq
"abc" == "ab*"              // TRUE
// string eq
"abc" == concat("ab", "*")  // FALSE

The above is basic math using an operator association and commutativity on literals.

With fields it becomes even more confusing:

// string eq
field == "abc"              // TRUE
field == concat("ab", "c")  // TRUE

// match eq
field == "abc*"             // TRUE
// string eq
field == concat("ab", "c*") // FALSE 

Which end up breaking transitivity:

// assume field == "ab*"
"ab" == "ab*"               // TRUE
"ab*" == field              // TRUE
"ab" == field               //FALSE

This is a big deal. Essentially == is not equals.
So why reuse the operator in the first place if it has different semantics?

Proposal

Introduce a separate operator, say ~= or e= (eql equals) that takes over the current == semantics, such as identifying the wildcard pattern and being case insensitive plus whatever other semantic we want to add moving forward.
This frees the == operator to mean exact equality and nothing else, just like the rest of the operators.

Pros:

• == means equals just like everybody expects. No gotchas, footnotes or trappy behavior
• == is aligned with the rest of the operators including >=, <=
• the custom EQL semantics are captured by a custom operator

Cons:

• breaking compatibility with existing queries. Requires 80-90% rewrites

[rw-access]

I surveyed the primary users (@elastic/security-intelligence-analytics)of EQL, and it was unanimous (12/12) that == "wild*card" is a feature and not a bug.

This was a deliberate design decision and is one that is expected and desired. Maybe the == operator isn't interpreted as literally as you're proposing. But there appears to be no cognitive dissonance by the people that are already using the I didn't ask ask for further analysis.

We can talk more about this offline, but I think == "some*wildcard" should be here to stay.

[costin]

A few more examples of == breaking down:

"a"  > "A"   // TRUE
"a" == "A"   // TRUE
"a" <= "A"   // FALSE
"a" in ("A")   // FALSE

"ab" == "a*"   // TRUE
"ab" <= "a*"   // FALSE
"ab" in ("a*") // FALSE

[jpountz]

I find the semantics of == to be surprising too. While I understand how important wildcard matching is for users of EQL, I share Costin's concerns around giving == semantics that do not meet the requirements of an equivalence relation and are also very different from major query languages.