Cannot include more than one script processor in ingest node pipeline

[andrask]

Elasticsearch version: 5.5.1

Plugins installed: []

JVM version (java -version):

openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

OS version (uname -a if on a Unix-like system):

Linux fce54823f0e7 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Ingest node pipeline containing successive script processors with inline definition executes and compiles only the last processor. The expected behavior is to compile and execute all scripts.

Steps to reproduce:

TEST_DOC='
{
  "pipeline": {
    "description" : "Enhance doc",
    "processors" : [
      {
        "script": {
          "inline": "invalid command that does not compile"
        },
        "script": {
          "inline": "ctx.field1 = '1'"
        },
        "script": {
          "inline": "ctx.field2 = '2'"
        }
      }
    ]
  },
  "docs" : [
    {
      "_index": "i",
      "_type": "line",
      "_id": "GGFsUYR_dur3woEnKsSA",
      "_score": 1,
      "_source": {
          "@timestamp": "2017-07-24T15:45:13.257Z"
      }
    }
  ]
}
'

curl -XPOST -f ${ES:need ES}/_ingest/pipeline/_simulate -d"$TEST_DOC" | python -m json.tool

What happens:

1. The pipeline runs successfully
2. The resulting doc contains only _source.field2

Expectations:

1. The first script should fail to compile.
2. After removing the first script, the resulting doc should contain the _source.field1 and _source.field2 keys.

[martijnvg]

@andrask What you have done here is trying to specify multiple scripts in a single processor. What you should do instead is specifying multiple processors each with a single script:

TEST_DOC='
{
  "pipeline": {
    "description" : "Enhance doc",
    "processors" : [
      {
        "script": {
          "inline": "invalid command that does not compile"
        }
      },
      {
        "script": {
          "inline": "ctx.field1 = '1'"
        }
      },
      {
        "script": {
          "inline": "ctx.field2 = '2'"
        }
      }
    ]
  },
  "docs" : [
    {
      "_index": "i",
      "_type": "line",
      "_id": "GGFsUYR_dur3woEnKsSA",
      "_score": 1,
      "_source": {
          "@timestamp": "2017-07-24T15:45:13.257Z"
      }
    }
  ]
}
'

ES should have returned an error, but a different one. Multiple keys with the same name in the same json object isn't allowed. @tlrx Do you know why this illegal json doesn't fail?

[andrask]

@martijnvg OOOOH, that explains. The error would be tremendous help, though.

[danielmitterdorfer]

@martijnvg I think the reason why this does not fail on 5.x is that we've turned on strict duplicate checks in XContent parsing only with 6.0 (see #22225) and I guess that the ingest APIs have no hand-coded duplicate checks?

[martijnvg]

@danielmitterdorfer Ok, I see. I somehow thought we did this to for 5.x. Then I think we can close this issue as of 6.0 we have the right behaviour when a request like in the issue description is sent.