provide data types for semver and semver_range to enable indexing and querying semantic version values

[geekpete]

Describe the feature:

Provide dedicated types to index and query against semantic versions with a semver and a semver_range type.

Use of Semantic Versioning is so widespread that a dedicated semver type and semver_range type will be useful primitives to add to Elasticsearch for varied use cases.

### semantic version type and semantic version range type example usage

PUT /semver-test
{
  "mappings": {
    "properties": {
      "message": "text",
      "version": {
        "type": "semver"
      },
      "compatible_versions": {
        "type": "semver_range"
      }
    }
  }
}

# example app1
POST /semver-test/_doc/1
{
  "app": "coolapp-ce",
  "version": "1.3.2",
  "compatible_versions": {
    "gte" : "1.0.0",
    "lte" : "1.4.0"
  }
}

# example app2
POST /semver-test/_doc/1
{
  "app": "coolapp-TNG",
  "version": "6.1.9",
  "compatible_versions": {
    "gte" : "6.0.0-alpha1",
    "lt" : "6.2.0"
  }
}

# example app3
POST /semver-test/_doc/1
{
  "app": "coolapp-final",
  "version": "6.2.1",
  "compatible_versions": {
    "gte" : "6.2.0-alpha1",
    "lt" : "6.3.0"
  }
}

# semver range query
GET /semver-test/_search
{
    "query": {
        "range" : {
            "version" : {
                "gte" : "6",
                "lte" : "6.2.0"
            }
        }
    }
}

# query against semver range type
GET /semver-test/_search
{
  "query": {
    "term": {
      "compatible_versions": {
        "value": "6.2.0-beta3"
      }
    }
  }
}

More advanced range combinations might be interesting to implement such as how Python's pip tool allows versions to be specified:
https://www.python.org/dev/peps/pep-0440/#version-specifiers

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[bczifra]

The discussion around integer/long mappings seems relevant to this issue.

[ejsmith]

Yes please!! This would be super helpful.

[webmat]

ECS captures versions in straightforward .version fields, in order to be able to capture whatever messy versions are out there in the wild. Semver are the "cleanest" versions, but there's also date-based versions (like Ubuntu's), there's composite versions like RHEL's package release versions & so on.

My ideal scenario would be to have a datatype we can add directly to existing .version fields as a multi-field, and would silently ignore what's not semver.

Here's a few things I'd like to see supported:

• depth from one numeric (1) to at least 3 (the typical 1.2.3), and ideally 4 (like Chrome's 78.0.3904.108)
• I'm split on whether we should specifically support labels (e.g. 1.2.3-beta1). Ignoring them and still indexing the numeric part would be great, as a start. If we can eventually support labels as well, even better.
• I would love to see range queries on the semver datatype, e.g. tls.version >= 1.2. For me this is the highest value we can get from this.
  • I'd love to see support for gt, gte, lt, lte as the basics
  • Bonus points if we can get semantically nicer operators akin to Rubygem's ~> 1.2.3 which essentially means >= 1.2.3 and < 1.3. But this can be accomplished with the 4 building blocks gt, gte, lt and lte. So less pressing.

I don't currently have a clear use case for semver_range, but this could also prove useful.

[jpountz]

Thanks @webmat, this is very useful. I agree ranges are important. I think we could support versions with the following restrictions:

• <numeric identifier>s for major, minor and patch can't be greater than 65535 (so that they can be stored on 2 bytes)
• <pre-release> is limited to 13 characters (can be stored on 10 bytes since there are only 38 allowed characters)

The build number doesn't need to be part of the indexed representation since it doesn't matter for ordering. This would allow to store versions as a 16 bytes integer, which is the maximum that Lucene supports. Do these restrictions feel ok?

[webmat]

I think the 2 bytes restriction per number and the overall 16 byte integer representation are reasonable, yes.

Looking at your comment before edits, I was about to say I didn't expect us digging into or interpreting the labels that much (the number in alpha.1). I would specifically avoid trying to parse labels, actually. There's conventions there, but people do weird things with these labels. You'll note that I'm not even calling them "pre-release", but specifically "label" which doesn't impose a semantic meaning. 2 examples to support this:

• Redhat's "Release" version strings, e.g. "5.el6"
• Even in projects following SemVer, I wouldn't be surprised to see 1.0.0-ga 🤷‍♂ :-)

Whenever we reference this part of a version string, I would suggest we name it "label" or something that doesn't impose a semantic meaning.

To confirm, is it correct to say that we would parse and index up to the first 3 numbers, but not the 4th (e.g. in longer versions such as Chrome's "78.0.3904.108")?

I'm good with this, as long as the 4 number versions do get parsed successfully. If someone is looking for an exact build number, it they should search for the exact match of the version string, not version.build:108

I'm also good with truncating the label at 13 bytes. Or were you suggesting 1.2.3-really-really-long-label would fail parsing?

[jpountz]

My worry is that the specification is very clear about how these labels should be used to compute precedence, and I worry that users will be caught by surprise if 1.0.0-alpha2 doesn't compare less than 1.0.0-rc1 or if 1.0.0-alpha.2 doesn't compare less than 1.0.0-alpha.10.

Regarding versions that have long labels, I believe we would be able to not reject them by storing the prefix in the index and the whole thing in doc values. Then range queries would use the index for the first 13 characters and fall back to doc values when version labels are longer, which should be rare.

How to handle non-semver version schemes is a good question. For instance, when there is no ambiguity, we could support additional schemes directly e.g. major.minor.patch.label (dot instead of dash) for Chromium versions. The Redhat version sounds more complicated to support to me since I guess we will want to support major.minor too, without patch and label information and this might introduce ambiguity. Or maybe this field should be more structured to give more flexibility, e.g. something like below instead of a single string. Note that everything would still get indexed in a single field.

"version": {
  "major": 5,
  "minor": 0,
  "patch": 0,
  "label": "el6"
}

[webmat]

Point no 9 of Semver states:

Pre-release versions have a lower precedence than the associated normal version.

Supporting that part makes sense and is straightforward. It states that all versions that include "a pre-release label" are lower precedence -- or earlier version -- than the same version without a pre-release label. 1.0.0-alpha < 1.0.0. I'm 👍 to support this, and I retract my insistence on calling this "labels"; calling this "prerelease" will give users a better mental model on how the sorting works.

But to me the spec seems to give up on specifying anything in the meaning of the acceptable labels, in terms of comparing one label to another. If you focus on the last two example labels they give:

Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92

These examples are actually all over the place. So I think working to support sensible sorting based on the label part of the version may end up being a lot of effort, for something that's not actually defined by SemVer 2.

I think a useful approach to handling the sorting of the labels would actually to also "give up" and just do lexical sorting.

Pros:

• simpler to implement
• simple to understand
• will "work" for all sorts of made-up label combinations or transitions between labeling strategies

Cons:

• has the typical problems of lexical sorting where "beta11" < "beta2"

Glancing at version numbers of various package lists like my workstation's, homebrew's or a Linux distro's is a good reminder of how narrow of a corner case sensible interpretation of the labels is. A significant percentage seem to not follow SemVer (maybe 10%?), the majority seems to follow a sensible major.minor.patch format, but few actually have labels; those that do, rarely seem to follow the a logic close to -[label][optional numeric].

I'd rather we spend our cycles on making version datatype support resilient to the craziness seen in the wild. Ideas:

1. Try semver parsing with prerelease labels having lower precedence than non labeled version, then lexical sorting of the labels
2. Fall back to parsing numbers separated by dots and ignoring what doesn't match ("1.2.3.4", "1.0beta4+r3088" => 1.0, "17.0.0.Final" => 17.0.0, "2005-02-20" => 2005.02.20)
3. Fall back to lexical sorting of the even crazier version strings ("00-5.0.5", "r2917")

I'm also ok if we don't implement a fall back like described above, as long as version parsing of an unsupported format doesn't fail indexing. That part is absolutely critical, IMO.

[jpountz]

But to me the spec seems to give up on specifying anything in the meaning of the acceptable labels, in terms of comparing one label to another.

I don't have the same reading. These sentences in particular are only about how versions with the same major, minor and patch should compare depending on their labels:

Precedence for two pre-release versions with the same major, minor, and patch version MUST be determined by comparing each dot separated identifier from left to right until a difference is found as follows: identifiers consisting of only digits are compared numerically and identifiers with letters or hyphens are compared lexically in ASCII sort order. Numeric identifiers always have lower precedence than non-numeric identifiers. A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal.

Let me try to think more about whether I can find a way to index versions that never fails yet still honors precedence rules of semver.

[webmat]

You're right, I overlooked point no 11, which does a better job at defining precedence between labels of a same version.

I think my point still stands on how much of a corner case sorting semver labels is, vs their "correct" usage in the wild, however.

[jpountz]

I think my point still stands on how much of a corner case sorting semver labels is, vs their "correct" usage in the wild, however.

I agree with this statement, but I know some people can be very picky about it. I also like the fact that we could just direct people to the semver docs to explain how precedence works vs. having to define our own rules.

I played a bit with encoding, it looks possible to accept arbitrary strings yet honor precedence rules for version that use the semver scheme. Precedence would also work as expected for variants like XX.YY (like Ubuntu) or X.Y.Z.B (like Chromium). That said I think it'd be good to enforce some simple rules, e.g.

• only printable ASCII characters
• a reasonable maximum length (e.g. 32)

[webmat]

Yes, I agree with these restrictions.

From the POV of ECS, this datatype would likely be used as a multi-field appended to the various pre-existing .version keyword fields. So I don't mind if we end up being pretty strict on this proposed datatype. If users are dealing with a package/library/service that has a crazy version scheme, they can always use the keyword field instead of its nicer .version.semver multi-field.

And thinking more about the crazy version strings, I guess they would kind of work anyway. The "r2917" above is a real version seen on Homebrew. If this parses to 0.0.0 + pre-release label "r2917", I guess the sorting will work as expected, when sorted against "r2916" and "r2918" 🤷‍♂ 🙂