[ML] Creating a new job fails if it's using a reindexed results index

[droberts195]

Steps to reproduce:

1. Install ES 5.x
2. Run a job
3. Do a full cluster restart update to ES 6.7
4. Install Kibana 6.7
5. In the migration assistant reindex all the indices created in 5.x into the 6.x format
6. Try to create a new job

The following error is the result:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "invalid_index_name_exception",
        "reason" : "Invalid index name [.ml-anomalies-shared], already exists as alias",
        "index_uuid" : "_na_",
        "index" : ".ml-anomalies-shared"
      }
    ],
    "type" : "invalid_index_name_exception",
    "reason" : "Invalid index name [.ml-anomalies-shared], already exists as alias",
    "index_uuid" : "_na_",
    "index" : ".ml-anomalies-shared"
  },
  "status" : 400
}

After reindexing the output of _cat/aliases is this:

alias                          index                                    filter routing.index routing.search
.kibana                        .kibana_1                                -      -             -
.ml-state                      .reindexed-v6-ml-state                   -      -             -
.ml-state-write                .reindexed-v6-ml-state                   -      -             -
.security                      .security-6                              -      -             -
.ml-anomalies-.write-farequote .reindexed-v6-ml-anomalies-shared        -      -             -
.ml-anomalies-farequote        .reindexed-v6-ml-anomalies-shared        *      -             -
.ml-anomalies-shared           .reindexed-v6-ml-anomalies-shared        -      -             -
.ml-notifications              .reindexed-v6-ml-notifications           -      -             -
.ml-annotations-read           .ml-annotations-6                        -      -             -
.ml-annotations-write          .ml-annotations-6                        -      -             -

Instead of checking whether the required indices exist, our code needs to check whether the required name exists as an index or alias. It should tolerate opening a job when .ml-anomalies-shared is an alias rather than an index.

It is critical that we fix this before 6.7.0 GA.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

@droberts195 looking a bit at this.

If the alias .ml-anomalies-shared exists, we obviously cannot add the an alias .ml-anomalies-.write-farequote2 that points to the alias. It will have to be to the concrete index behind it. I am starting to dig into where this is all happening.