Watcher: 5.x/6x Upgrade breaks with watches with stored scripts and lang field

[spinscale]

Elasticsearch version (bin/elasticsearch --version): 6.x

When a user adds a watch in 5.x, that contains a script and the script section contains a lang field, the watch will be broken after upgrading to 6.x. The original PR adding this check was done in #25610

Reproduction

Have a 5.x node where you add this script and this watch (or just index via bulk under 6.x as shown as well).

POST _scripts/my_script
{
  "script": {
    "lang": "painless",
    "source": "return true"
  }
}

# add under 5.x
PUT _xpack/watcher/watch/my_watch
{
  "trigger": {
    "schedule": {
      "interval": "10h"
    }
  },
  "input": {
    "simple": {
      "foo": "bar"
    }
  },
  "condition": {
    "script": {
      "id": "my_script",
      "lang": "painless"
    }
  },
  "actions": {
    "logme": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}


# or emulate under 6.x
POST _xpack/watcher/_stop

PUT _bulk
{"index":{"_index":".watches","_type":"doc","_id":"my_watch"}}
{"trigger":{"schedule":{"interval":"10h"}},"input":{"simple":{"foo":"bar"}},"condition":{"script":{"id":"my_script","lang":"painless"}},"actions":{"logme":{"logging":{"text":"{{ctx}}"}}}}

POST _xpack/watcher/_start

# this is gonna fail
GET _xpack/watcher/watch/my_watch

with the following exception

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "illegally specified <lang> for a stored script"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "illegally specified <lang> for a stored script"
  },
  "status": 400
}

If you store the above watch in 5.x you will get a depreciation message though.

Discuss: Would it make sense to check for the lang field in the condition, the transform and the conditions and transforms in the action, and optionally remove the lang field as part of the reindex that is done when running the upgrade API?
This will not catch all the possible cases, but I think 99%.

[elasticmachine]

Pinging @elastic/es-core-infra

[spinscale]

we talked about this and this is our way forward:

We will fix the upgrade helper to also take care of this. Currently the upgrade helper just removes the _status by using a script in the reindex action.

This script needs to be extended to do the following

• Check if the condition is a script based one. If true check if the script is using a lang parameter and optionally remove this parameter
• Check if the transform is a script based one. If true check if the script is using a lang parameter and optionally remove this parameter
• For each action:
  • If condition is script based and check and remove lang parameter
  • If transform is script based and check and remove lang parameter

This also requires a 5.x backport.

The only possibility left over is a chain transform, which we could cover as well, but I'd start with the above and then check how this works. See https://www.elastic.co/guide/en/elastic-stack-overview/6.4/transform-chain.html