[ML] Allow users to annotate ML anomaly results

[droberts195]

It would be nice if a user who knows the reason for an interesting feature in the ML results to be able to annotate this.

To enable this we could add a new annotation result type, similar to this:

    {
      "job_id": "it-ops-metrics",
      "result_type": "annotation",
      "timestamp": 1454944200000,
      "end_timestamp": 1454946000000,
      "annotation": "Datacenter was isolated for failover testing",
      // The following is optional
      "detector_index": 1
    }

By making the detector index optional, the annotation can apply to either the whole job or just a specific detector.

Unlike other ML results, instead of timestamp and bucket_span annotations have timestamp and end_timestamp so that the annotation can span an arbitrary time period.

Originally it was thought that the same functionality could be used by the ML C++ code to add reasons why it created results, but the current thinking is that it is better to have separate functionality for the two use cases, hence elastic/ml-cpp#197 has been raised to discuss labelling by the C++ code.

[elasticmachine]

Pinging @elastic/ml-core

[tveasey]

We should think about structured input the user can provide here as well, the simplest thing would be something to the effect of "this is important" or "this is not important". It is much harder to infer things from free text input if we start using this to provide supervision to our models (although having both is no bad thing).

[richcollier]

maybe this is also a solution for this ER? https://github.com/elastic/enhancements/issues/3322

[droberts195]

We should think about structured input the user can provide here as well

We talked about this feature in the roadmap discussion in Dublin and decided that initially this will just be annotations by humans for human consumption.

If we ever add user feedback to provide supervision to our models then we can either extend the format or add a new result type to store this feedback separately.

[droberts195]

Some possibilities based on a discussion with @peteharverson are:

1. Allow global annotations, i.e. no job_id field
2. Allow annotations that apply to multiple jobs, i.e. job_id is an array with multiple values
3. Allow annotations that apply to a job group, i.e. allow job_group to be specified instead of job_id

If we allow groups we need to decide how and when they'll be expanded to specific job IDs.

This makes the deletion logic quite complicated - annotations should be deleted when all jobs they relate to are deleted, but the wide variety of ways of specifying which jobs an annotation relates to makes determination of this complex.

[sophiec20]

I wonder if detector_index might also be a optional one?

[droberts195]

I wonder if detector_index might also be a optional one?

We discussed this and decided to have detector_index as the only option, instead of by/partition field values.

[droberts195]

We also decided that it would be useful to have persistent annotations, that don't get deleted if all the jobs they relate to get deleted. This would be useful for the case of repeatedly deleting and recreating a job to try out slightly different configurations.

So there should be an is_persistent boolean field in the annotations too.

[droberts195]

The other thing we discussed is editing annotations. It will be incredibly frustrating for users if they are not allowed to edit annotations, for example, after making a typo.

This leads on to the question of who can edit which annotations. Ideally there would be object level security such that each user could only edit annotations they added, unless the owning user opened up the permissions to more users. However, we do not currently have object level security. (The same problem applies to job configs for example.)

We should not introduce a form of object level security purely for annotations. Therefore we decided that if you can edit annotations you can edit any annotations.

There is also the question of whether creating and editing annotations should be an admin or user level privilege. Most write actions require admin privileges. However, that would cripple annotations as it's likely they'll be most useful to people looking at the results, and these people do not require admin privileges. Therefore we decided that the machine_learning_user role should allow adding and editing of annotations. (The actions could still be admin actions, but the machine_learning_user role definition should incorporate them. Alternatively the actions could be "monitor" actions, in which case no changes would be required to the definition of machine_learning_user. This is a low level implementation decision that can be deferred.)

[walterra]

Do we want to support additional metadata similar to timestamp/end_timestamp, for example a value range? This would allow annotations based on time and value like in this demo: