Enrich policy execution can be a heavy burden on the master

[DaveCTurner]

Today when an enrich policy is executed, building a new copy of the enrich index, we use the elected master to coordinate the underlying reindex task which operates in batches of 10,000 documents by default. A user reported that a particular enrich policy execution would reliably cause their master to fail with an OutOfMemoryError. The policy in question was of geo_match type and the documents, containing geoshapes, could be quite large. Their master had a 1GB heap, which is appropriate for their small and well-run cluster, but it appears it simply could not hold 10,000 geoshapes in memory on the master at once.

As a workaround they reduced the (undocumented?) enrich.fetch_size to 5000, which was enough to avoid the OOM, but I'm still concerned about the strain this puts on the master.

A few ideas for possible improvements:

• Move the coordination of the reindexing job onto a different node.
• Strengthen a circuit-breaker to prevent an OOM like this one.
• Adapt the reindex batch size to the resources available.

I think we should definitely do the first; the other two are harder.

See https://discuss.elastic.co/t/why-does-an-enrich-policy-get-executed-on-the-master-node/263241 for more details.

/cc @consulthys

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[consulthys]

Thank you so much @DaveCTurner !!

[martijnvg]

Thanks for filing this bug @DaveCTurner and @consulthys for reporting this problem in the forum. I think we have underestimated the resources it takes to rebuild the policy index with geo shapes.

Move the coordination of the reindexing job onto a different node.

Yes, I think we should migrate the enrich policy execution to use the persistent task framework, so that these operations (for any policy type) get executed on other nodes (anything but master nodes).

[martijnvg]

Instead of migrating the enrich policy execution over to persistent tasks, the enrich policy execution can be refactored that just perform the execution policy on another non master node. In particular that part that is encapsulated in EnrichPolicyRunner class. Coordination and delegation would still be done on the elected master node.

This is relatively easy to implement and avoids making a breaking changes. The enrich policy execution api returns a node task id if wait_for_complete=false and if policy execution is refactored to use persistent tasks we would not reliably be able to include a node task id (ideally we wouldn't want to do this, but instead return persistent task id, but we would need to keep returning a node task id for bwc). Also the enrich stats api should be refactored to not return a node task id, but a persistent task id. Irregardless of this for 8.0, I do think we should change the enrich policy execution api to use persistent task framework.