Skip to content

ingest: only compile templates if the value is templated #37120

@jakelandis

Description

@jakelandis

Many of the ingest node processors support template snippets (mustache). Each field that supports template snippets requires compilation the first time the pipeline is used.

Elasticsearch has circuit breaker such that no more than 75 scripts per 5 minutes may be compiled. Every script (not only ingest node scripts) is subject to this (configurable) limit. A moderate number of processors can trip this limit, even if no templated values are defined.

For example, a single set processor always counts as 2 compilations since the field and value both support template snippets. The compilation happens even if the field and value do not define a {{ }} template.

This issue is to address the unnecessary compilation for fields that support template snippets. The goal to reduce the number of compilations to only the minimum number actually required.

This issue is more likely on newer versions of Elasticsearch since in 6.4 additional fields were added to support template snippets and 6.5 the if condition was added, which also requires a script complication (if defined).

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Data Management/Ingest NodeExecution or management of Ingest Pipelines including GeoIP

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions