Limit request size

[danielmitterdorfer]

Overly large bulk request can threaten the stability of Elasticsearch. Hence we want to limit the size of a bulk request.

Although the need originally arose for bulk requests, the solution will apply to requests in general and not just bulk requests.

There will be two (configurable) limits:

• A limit on the size (in bytes) of an individual request
• A limit on the size (in bytes) of all requests that are currently in flight

[dadoonet]

Just a thought: I wonder if the default limit should be something like the default http.max_content_length divided by two or something along those lines?
May be we should (also?) check that this max size is actually not bigger than http.max_content_length and reject the settings in such a case?

[danielmitterdorfer]

Thanks for your input @dadoonet! I am not sure about your suggestion on the check of http.max_content_length though as this relies on a protocol level setting whereas I'd try to enforce bulk request size in a protocol-independent fashion. I mean this should also work if somebody uses only the transport protocol and then I wouldn't do a validation of this setting based on a http related one. But tbh I did not look into the code yet. wdyt?

[dadoonet]

ha! That is correct. I thought http.max_content_length was applied to transport layer as well...
Ignore me then! :D

[danielmitterdorfer]

Ok, thanks for the clarification.

[danielmitterdorfer]

Design Parameters

After looking at the source, I have identified a few options which I'd like to discuss before implementing this feature. Below are some design parameters which are worth considering from my point of view.

Type of endpoint

A (bulk) request can hit the cluster via the following endpoints:

1. HTTP: hint: the REST handler reissues the HTTP request as a transport request via the client interface.
2. Transport (or more precise: remote transport)
3. Local transport: An optimization to avoid the overhead of a remote call when just forwarding a request within the same process.

Applicability of the limit

This means whether we want to apply the limit for all types of requests or just for a limited number of explicitly defined request types (like bulk requests).

When to check the limit

• Apply during deserialization: This would allow us to abort request handling as soon as we know the limit is hit but at the expense of implementing this for all protocols separately. For the transport protocol, the right place looks like MessageChannelHandler#handleRequest().
• Apply before a (transport) request is handled (i.e. in the appropriate subclass of TransportAction): This would simplify the validation as we would just check once for all endpoint types (as we know that HTTP requests are reissued as transport requests). However, at this point in time the request is already deserialized thus consuming memory and all we can do is preserve any additional system resources by aborting request processing.

Request size calculation

To know whether a request has (b)reached the limit, we need to calculate its size. Considering BulkRequest for now, we have two options on how to determine the request size:

• BulkRequest#estimatedSize(): This works but has shortcomings in corner cases (see e.g. BulkRequest estimated size in bytes does not include script param size #15589)
• If we enforce the limit already during deserialization we can calculate request size exactly, e.g. by writing a custom StreamInput implementation that counts the bytes deserialized.

Proposal

Based on the options above, I want to sketch a simple solution proposal:

Considering that it is likely we want limit checks not only for bulk requests but also similar ones, like multi-gets, we should not tie this too specifically to bulk requests.

Hence, each for each request type that should be size-limited, the corresponding TransportRequest class implements a new interface RequestSizeEstimator (draft name):

public interface RequestSizeEstimator {
  int getRequestSizeInBytes();
}

We define one configurable request size limit (default e.g. 50MB) for all request types and implement limit breach detection in a high-level fashion as an ActionFilter. This ActionFilter checks whether the TransportRequest implements RequestSizeEstimator and checks against this limit. If the limit is breached, we throw an appropriate exception indicating a client error (4xx in HTTP speak).

In summary, the pros and cons of this solution are:

Pros:

• The limit is only checked in a single place in the code base
• Easily extensible: We can add checks for new request types easily by having the corresponding transport request class implement RequestSizeEstimator.

Cons:

• We deserialize the full request before we check the limit. This means we still hit a potential memory spike but we avoid using any other resources (CPU, I/O) after that point.
• The size calculation may not be entirely accurate (see my comments above).

Feedback is very much appreciated.

[danielmitterdorfer]

Based on feedback I got by @bleskes, here's a revised approach:

We limit the size of requests on protocol level during deserialization. We consider two cases:

• Receiving a request: We will apply one global limit for all requests that are in flight
• Sending a request (internally in the cluster): We will apply a limit for each individual request

Limiting on protocol level has a couple of advantages:

• We cancel request processing as soon as we hit the configured memory limit
• It allows us to introduce a limit per action (not in the scope of this ticket though)

In the first step, we will implement an action-independent limit which applies to all actions not just bulk actions.

Transports that need to be considered are Netty transport and local transport (for testing purposes). HTTP is unaffected because we just stream data from the HTTP layer to the transport layer (no up-front allocation).

We need to check whether circuit breakers are a feasible option to detect limit breaches.

[danielmitterdorfer]

@dakrone: Could you share your thoughts whether using CircuitBreaker is feasible for limiting memory usage in the following two scenarios?

1. Limiting the memory usage of all in-flight requests on a single node level.
2. Limiting the memory usage of each individual outgoing request of a node.

In both cases the memory usage will be determined by the size of the deserialized request payload.

(more details above; that's just the summary for you)

Scenario 1

I have seen that there is already a "request" circuit breaker but its BreakerSettings are (a) cluster-wide and (b) relative to the amount of available heap memory. I'd tackle this by introducing new BreakerSettings for all requests in flight at node level (which would have to be a new value in org.elasticsearch.common.settings.Settings.Scope)) and with an absolute but user-configurable limit (e.g. 50mb).

Scenario 2

We also want to limit memory usage on a single request basis (not across all requests that are in flight) and tbh I think that circuit breakers are not a good fit because we would need a new circuit breaker instance for each request as far as I understand it. However, it would make the implementation more uniform.

It would be great to hear your input on this.

[dakrone]

@danielmitterdorfer

Could you share your thoughts whether using CircuitBreaker is feasible for
limiting memory usage in the following two scenarios?

Limiting the memory usage of all in-flight requests on a single node level.
Limiting the memory usage of each individual outgoing request of a node.

I have seen that there is already a "request" circuit breaker but its
BreakerSettings are (a) cluster-wide and (b) relative to the amount of
available heap memory. I'd tackle this by introducing new BreakerSettings for
all requests in flight at node level (which would have to be a new value in
org.elasticsearch.common.settings.Settings.Scope)) and with an absolute but
user-configurable limit (e.g. 50mb).

This is exactly what the circuit breaker is for, you can easily define a new
breaker (and breaker type) in HierarchyCircuitBreakerService to do just that.
I would recommend this heartily :) (and I'm happy to help)

We also want to limit memory usage on a single request basis (not across all
requests that are in flight) and tbh I think that circuit breakers are not a
good fit because we would need a new circuit breaker instance for each request
as far as I understand it. However, it would make the implementation more
uniform.

To me, this doesn't sound like a good fit for the circuit breaker (since it is
designed to be per-node, not per-request). Instead, this sounds like a better
fit for the validation framework, since that works on the per-request level.