Replace hand parsing of requests/queries/mappings/etc with a grammar

[clintongormley]

Requests/queries/etc currently have custom parsers, which make it more difficult to be certain that they are error free. We should replace all this hand parsing with some form of grammar, which defines exactly what syntax is allowed. This will also benefit those implementing clients, as they will have a grammar that can be interrogated programmatically.

Any unknown parameters should throw a parsing exception.

We can add custom rules where needed (eg to disallow certain combinations of parameters like using fuzzy in the multi_match query with type cross_fields).

[nik9000]

Requests/queries/etc currently have custom parsers, which make it more difficult to be certain that they are error free. We should replace all this hand parsing with some form of grammar, which defines exactly what syntax is allowed.

Cool.

This will also benefit those implementing clients, as they will have a grammar that can be interrogated programmatically.

That'd only really work if you were careful with how the grammar was written - it can't embed any Java code.

We can add custom rules where needed (eg to disallow certain combinations of parameters like using fuzzy in the multi_match query with type cross_fields).

And (probably) none of these would work with just the grammar but that is pretty OK. If you wrote them in a constraint style clients could probably generate code for them.

[lukas-vlcek]

Happy to see ES is going to get DSL grammar! Are there already any public information about the grammar?

[lukas-vlcek]

@nik9000 I am pretty sure well written grammar can check any types of rules. Grammars can contain a code, for example JavaScript code (have you seen PEG.js?). When compiled I can imagine this could be still performant.

[Mpdreamz]

Much of the work we plan to do for NEST 1.5 is revisit ALL of our of our mappings of the elasticsearch responses and requests to figure out what we are missing and hopefully get to write "a spec" for it.

We are currently thinking to write these down using json schema draft 4 which is not going to be able to express 100% of the constraints but it'll cover a great deal of it.

The upside for the clients is that the json schema is that it'll describe POJO's that can be used both for parsing and generating json. The downside is that using POJO jackson databinding (or the equivalent in other langs) is slightly slower then using the stream API. Which is most likely why elasticsearch went with with the manual stream parsing/writing in the first place.

In the .NET client we currently use databinding on POCO's but would hope to be able to generate serialization AND deserialization using a streaming API from the json schema's without having to do the tedious bits ourselves.

A fullblown lexical parser for elasticsearch will be awesome on elasticsearch side not entirely sure it'll help clients since elasticsearch needs are the dual of clients needs.

client writes => elasticsearch reads
client reads <= elasticsearch writes 

So just to be sure are we talking about yacc/antlr/ragel style grammars here or codifying contracts using types/schemas?

[nik9000]

@nik9000 I am pretty sure well written grammar can check any types of rules. Grammars can contain a code, for example JavaScript code (have you seen PEG.js?). When compiled I can imagine this could be still performant.

Embedding code makes it not language portable which isn't great clients. You want something like the json schema draft @Mpdreamz mentioned and a tool to do some code generation from it.

[clintongormley]

I've had a go at spec'ing out the match query using JSON-schema. The result is not 100% accurate, but not bad:

    {
      "id": "http://schema.elasticsearch.org/query",
      "title": "Schema for all queries",
      "type": "object",
      "default": {
        "match_all": {}
      },
      "max_properties": 1,
      "properties": {

        // list all query names here, plus their { "field": "query"} vs 
        // the more advanced { "field": { "query":..}} forms
        "match": {
          "type": "object",
          "min_properties": 1,
          "max_properties": 2,
          "properties": {
            "_name": {
              "$ref": "#/definitions/params/query_name"
            }
          },
          "pattern_properties": {
            "^\\S+$": {
              "one_of": [
                {
                  "$ref": "#/definitions/params/query"
                },
                {
                  "$ref": "#/definitions/queries/match_query"
                }
              ]
            }
          }
        }
      },

      "definitions": {
        "queries": {

          // Advanced form of match query defined here
          "match_query": {
            "one_of": [
              {
                "type": "object",
                "required": [
                  "query"
                ],
                "properties": {
                  "query": {
                    "$ref": "#/definitions/params/query"
                  },
                  "type": {
                    "enum": [
                      "boolean"
                    ]
                  },
                  "operator": {
                    "$ref": "#/definitions/params/operator"
                  },
                  "minimum_should_match": {
                    "$ref": "#/definitions/params/minimum_should_match"
                  },
                  "analyzer": {
                    "$ref": "#/definitions/params/analyzer"
                  },
                  "boost": {
                    "type": "number"
                  },
                  "cutoff_frequency": {
                    "$ref": "#/definitions/params/int_or_float"
                  },
                  "fuzziness": {
                    "$ref": "#/definitions/params/fuzziness"
                  },
                  "fuzzy_rewrite": {
                    "$ref": "#/definitions/params/rewrite"
                  },
                  "lenient": {
                    "$ref": "#/definitions/params/bool"
                  },
                  "max_expansions": {
                    "$ref": "#/definitions/params/max_expansions"
                  },
                  "prefix_length": {
                    "$ref": "#/definitions/params/postive_int"
                  },
                  "zero_terms_query": {
                    "$ref": "#/definitions/params/zero_terms"
                  }
                }
              },
              {
                "type": "object",
                "required": [
                  "query"
                ],
                "properties": {
                  "query": {
                    "$ref": "#/definitions/params/query"
                  },
                  "type": {
                    "enum": [
                      "phrase",
                      "phrase_prefix"
                    ]
                  },
                  "analyzer": {
                    "$ref": "#/definitions/params/analyzer"
                  },
                  "boost": {
                    "type": "number"
                  },
                  "fuzziness": {
                    "$ref": "#/definitions/params/fuzziness"
                  },
                  "fuzzy_rewrite": {
                    "$ref": "#/definitions/params/rewrite"
                  },
                  "lenient": {
                    "$ref": "#/definitions/params/bool"
                  },
                  "max_expansions": {
                    "$ref": "#/definitions/params/max_expansions"
                  },
                  "prefix_length": {
                    "$ref": "#/definitions/params/postive_int"
                  },
                  "slop": {
                    "$ref": "#/definitions/params/postive_int"
                  },
                  "zero_terms_query": {
                    "$ref": "#/definitions/params/zero_terms"
                  }
                }
              }
            ]
          }
        }
      },

      // Each query param defined here (and reused by all other queries)
      "params": {
        "analyzer": {
          "type": "string",
          "pattern": "^\\S+$"
        },
        "bool": {
          "any_of": [
            {
              "type": "boolean"
            },
            {
              "enum": [
                "true",
                "false",
                "TRUE",
                "FALSE",
                "0",
                "1",
                ""
              ]
            },
            {
              "type": "integer",
              "minimum": 0,
              "maximum": 1
            },
            {
              "type": "null"
            }
          ]
        },
        "fuzziness": {
          "any_of": [
            {
              "type": "string",
              "pattern": "^(0|1|2|AUTO|1.0|0.\\d+)$"
            },
            {
              "type": "integer",
              "minimum": 0,
              "maximum": 2
            },
            {
              "type": "numeric",
              "minimum": 0,
              "maximum": 1
            }
          ]
        },
        "int_or_float": {
          "any_of": [
            {
              "type": "string",
              "pattern": "^\\d+(\\.\\d+)?$"
            },
            {
              "type": "number",
              "minimum": 0
            }
          ]
        },
        "max_expansions": {
          "default": 50,
          "any_of": [
            {
              "type": "string",
              "pattern": "^(\\d+)$"
            },
            {
              "type": "integer",
              "minimum": 0
            }
          ]
        },
        "minimum_should_match": {
          "any_of": [
            {
              "type": "string",
              "pattern": "^((100|\\d\\d)%|\\d+)$"
            },
            {
              "type": "integer",
              "minimum": 1
            }
          ]
        },
        "operator": {
          "enum": [
            "or",
            "and",
            "OR",
            "AND"
          ]
        },
        "positive_int": {
          "default": 0,
          "any_of": [
            {
              "type": "string",
              "pattern": "^(\\d+)$"
            },
            {
              "type": "integer",
              "minimum": 0
            }
          ]
        },
        "query": {
          "type": "string",
          "pattern": "\\S"
        },
        "query_name": {
          "type": "string",
          "pattern": "^\\S+$"
        },
        "rewrite": {
          "any_of": [
            {
              "enum": [
                "constant_score_auto",
                "constant_score_boolean",
                "constant_score_filter",
                "scoring_boolean"
              ]
            },
            {
              "type": "string",
              "pattern": "^top_terms_(boost_)?\\d+"
            }
          ]
        },
        "zero_terms_query": {
          "enum": [
            "none",
            "all"
          ]
        }
      }
    }

Issues:

• The match query accepts a single field name (defined as a pattern) and an optional key called _name. It is not possible to enforce this in JSON schema (currently)
• It'd be nice to say "you can't set max_expansions unless you're using fuzziness - can't do that either
• While we can express that eg minimum_should_match only applies when type == boolean, I can see that it'd be difficult to throw an error message explaining that the reason minimum_should_match is being rejected is because of the type setting.
• We'd like to be able to mark certain parameters as accepted-but-deprecated, which currently isn't possible.
• It'd be nice to extend a definition overriding just (eg) the default value, rather than repeating the whole defn.

Ideas:

• Perhaps a dependencies keyword which allows us to specify certain post-validation rules (like type: phrase doesn't accept minimum_should_match, or max_expansions only makes sense if you use fuzziness). This could be extended to include nice error messages.
• Add a deprecated flag, with an error message, eg "Deprecated in favour of Foo"
• While eg the fuzziness parsing could be implemented in generic JSON-schema style, we could also provide an optimized Java-friendly parser for some or all parameters

[tikitu]

I've a wee bit of experience with JSON-schema in python: you may indeed find that it's difficult to get quality error reporting, especially as you get into complex constraints. (You can express a lot with combinations of anyOf and allOf, but once you use anyOf the failure reports aren't much good.) I don't know if the same holds for implementations in other languages, but I would expect it. (The problem is quite general with anyOf: the grammar gives you no way to say "once we've seen this structure, commit to this alternative: if we fail deeper in the structure, report that this alternative failed." Instead, it has to backtrack and can only report that none of the alternatives succeeded.)

[tikitu]

I did some thinking a while back about approaching this from the opposite end. If you annotated the fields on an XXX(Query/Filter)Builder you could generate a good portion of the code of the corresponding Parser, and similar annotation processors could spit out e.g. Haskell code, or a JSON-schema spec, or whatever was needed for a different client or use case. (I was thinking of the Sense autocomplete database; obviously docs work from here too.) You can even mix in custom code, if the annotation processor is a bit smart. In that approach you aim for say 80% coverage in the annotations, and catch the more complex validation requirements in custom code. But because the annotations are java code, if you've got repeating patterns you're free to extract and name them, no matter how complex they are.

[clintongormley]

@tikitu yeah - your comment about error reporting is the thing that makes me most nervous. Also i was thinking that we don't just want to validate the input JSON, we actually want to extract the values and add them to a settings object for the query/filter/endpoint etc.

I think JSON schema as it is today is not sufficient, but it probably gives us the starting point for our own custom grammar.

[tikitu]

I'm not convinced that what you want is a grammar. I think maybe what you want is something from which you can auto-derive a grammar, but which also makes it easier to auto-derive other things. Seems likely to me that if you have a grammar you won't use it directly in java code, but rather use it to (mostly-semi-)generate code that gets checked in (and maybe tweaked, for performance or for whatever validation the grammar can't express). Same sort of story for client libraries: if I'm writing a library for a strongly typed language I want intermediate-structure type-safety which the JSON spec won't give me, so I'll use the spec to generate code but add my own type information rather than feeding the spec directly into a json-parsing library.