ingest-attachment support for per document `indexed_chars` limit

[dadoonet]

Coming from this discussion: https://discuss.elastic.co/t/how-to-control-the-indexed-chars-value-on-a-ingest-attachment-pipeline/123073/4

We today support a global indexed_chars processor parameter. But in some cases, users would like to set this limit depending on the document itself.
It used to be supported in mapper-attachments plugin by extracting the limit value from a meta field in the document sent to indexation process.

Here is my proposal.
We should add an option like reading this limit value from the document itself by adding a setting like indexed_chars_field.

Then we could do something:

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information. Used to parse pdf and office files",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "indexed_chars_field" : "size"
      }
    }
  ]
}

Then index either:

PUT index/doc/1?pipeline=attachment
{
  "data": "BASE64"
}

Which will use the default value (or the one defined by indexed_chars)

Or

PUT index/doc/2?pipeline=attachment
{
  "data": "BASE64",
  "size": 1000 
}

I'll propose hopefully soon a PR for it unless someone in the meantime reject that feature request or propose another implementation for it.

[jasontedor]

FYI @elastic/es-core-infra.

[jasontedor]

@martijnvg @talevy Do you have any thoughts on the proposal?

[talevy]

I do not have any objections, I am just slightly annoyed that one has to use an actual field in the document to be used as a parameter to the plugin. It would be nice if there were a way to pass _meta-like fields per-document. @dadoonet, would it make sense to keep this size field as part of the source? or would you expect the attachment processor remove it

[dadoonet]

I think the user can then apply a remove processor if needed.
Agreed that have meta fields would be better but do we have that infrastructure in ingest? If not, does it worth it to add for this feature?

[talevy]

@dadoonet cool. No meta fields respected, as far as I know. This would have to be a feature of ES-Core that would probably only be leveraged by ingest-pipelines (so I have a hard time seeing this justified for now).

[martijnvg]

I think @dadoonet approach makes sense. I only would name field_indexed_chars indexed_chars_field instead.

[dadoonet]

Agreed @martijnvg. I updated the description of the issue to reflect your name change proposal.