Default network buffer size causes higher GC pressure than necessary

[danielmitterdorfer]

Netty4HttpServerTransport uses the settings http.netty.receive_predictor_min and http.netty.receive_predictor_max to provide a properly configured RecvByteBufAllocator implementation to Netty. Their default value is controlled by the setting transport.netty.receive_predictor_size which varies between 64 kb and 512 kb (per allocated buffer).

The before-mentioned allocator is responsible for allocating memory buffers when handling incoming network packets and Netty will allocate one buffer per network packet.

We have run comparative benchmarks (nyc_taxis) track, once locally (i.e. via loopback) and once distributed (i.e. via a Ethernet) and analyzed the allocation behavior of Elasticsearch.

Network Connection Type	Bytes allocated outside of TLABs on network layer
Loopback	~ 78 GB
Ethernet	~ 2.13 TB

Note: On this particular machine transport.netty.receive_predictor_size was 512kb.

The root cause seems to be related to MTU (which differs greatly between loopback and regular network devices (65536 vs. 1500)). A smaller MTU leads to more network packets (but the buffer size stays the same) thus leading to more GC pressure.

In a custom build of Elasticsearch we set http.netty.receive_predictor_min to 5kb and http.netty.receive_predictor_max to 64kb and got comparable allocation behavior between local and distributed benchmarks.

Note: Our analysis focused only Netty4HttpServerTransport for a single Elasticsearch node. It is expected that Netty4Transport exhibits similar behavior and we should change the buffer size there too.

[danielmitterdorfer]

To find a "good" new default value, I ran our whole benchmark suite against an instrumented build that records the number of bytes read on the network layer.

Each of the graphs below show a histogram of the number of bytes read during network packet handling for the respective benchmark.

If we settle on just a single value, I'd tend to set the default buffer size to 32kb as this should satisfy most allocation requests while not wasting too much memory. I will run further benchmarks to verify it.

[jpountz]

Should we move forward on this? It looks like a low hanging fruit that has the potential to make things significantly better for most users?

[danielmitterdorfer]

@jpountz I totally agree. I ran a lot of benchmarks over the last few days (until today) and came to the conclusion that 32kb is indeed a good value. I just wanted to ensure that we don't have unwanted side effects.

Expect a PR within the next hour.

[jasontedor]

This is really great work @danielmitterdorfer, thank you for digging into this.

[danielmitterdorfer]

Thanks @jasontedor. I also checked our benchmarks and in almost all cases we see an upward trajectory. However, we have still one problematic case: outside of TLAB allocations for smaller heap sizes (2GB) for the full-text benchmark increased significantly causing a drop in indexing throughput. I am currently investigating what's causing this and running further tests. So: more digging ahead.

[danielmitterdorfer]

The investigation is not yet finished but I want to present a few intermediate results.

I ran the following benchmark:

esrally --pipeline=benchmark-only --track=pmc --challenge=append-no-conflicts-index-only --target-host=192.168.2.2:9200

and varied heap size (-Xms, -Xmx) between 1GB and 16GB as well as http.netty.receive_predictor_size from 16kb to 64kb.

Target machine specs:

• CPU: Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz
• 64GB RAM
• SSD
• JDK 1.8.0_112-b15
• OS Arch Linux with Kernel 4.8.7-1-ARCH
• 1 GBit Ethernet

I also attached flight recorder and produced GC logs for every trial run.

Here is a summary of the allocation statistics:

Note: object count, maximum object size, minimum object size, average object size, allocation rate for objects are metrics for allocations that happen outside of TLABs.

heap size [GB]	receive predictor size [KB]	median indexing throughput [docs/s]	object count	maximum object size	minimum object size	average object size	total memory allocated	allocation rate for objects	TLAB count	Maximum TLAB size	Minimum TLAB size	Average TLAB size	Total Memory Allocated for TLABs	Allocation Rate for TLABs
1	64	419	64262780	46.57 MB	16 bytes	3.07 kB	188.15 GB	135.44 MB/s	1080267	30.87 MB	152 bytes	280.91 kB	289.40 GB	208.33 MB/s
2	32	OOMEd	36108455	34.29 MB	16 bytes	1.89 kB	65.11 GB	113.81 MB/s	238169	31.48 MB	56 bytes	475.63 kB	108.03 GB	188.83 MB/s
2	64	1176	726691	46.57 MB	16 bytes	228.74 kB	158.52 GB	298.95 MB/s	689016	32.65 MB	232 bytes	433.96 kB	285.15 GB	537.75 MB/s
4	16	1195	683635	46.57 MB	16 bytes	182.84 kB	119.20 GB	231.08 MB/s	681134	28.87 MB	968 bytes	444.24 kB	288.57 GB	559.40 MB/s
4	32	1228	683848	46.57 MB	16 bytes	185.89 kB	121.24 GB	239.21 MB/s	707107	34.79 MB	328 bytes	425.68 kB	287.06 GB	566.40 MB/s
4	64	1223	758859	46.57 MB	16 bytes	224.56 kB	162.52 GB	319.36 MB/s	786162	26.51 MB	1.13 kB	376.33 kB	282.15 GB	554.45 MB/s
8	16	1215	682482	46.57 MB	16 bytes	184.81 kB	120.28 GB	231.39 MB/s	686732	30.01 MB	672 bytes	441.18 kB	288.94 GB	555.84 MB/s
8	32	1219	681496	46.57 MB	16 bytes	184.78 kB	120.09 GB	232.39 MB/s	674428	29.07 MB	776 bytes	449.35 kB	289.01 GB	559.25 MB/s
8	64	1185	778077	46.57 MB	16 bytes	216.13 kB	160.38 GB	308.89 MB/s	790786	32.90 MB	728 bytes	377.13 kB	284.42 GB	547.80 MB/s
16	16	1210	685745	46.57 MB	16 bytes	184.82 kB	120.87 GB	230.90 MB/s	689624	27.57 MB	1.18 kB	436.05 kB	286.78 GB	547.86 MB/s
16	32	1199	687613	46.57 MB	16 bytes	182.69 kB	119.80 GB	230.36 MB/s	695341	28.34 MB	264 bytes	437.61 kB	290.19 GB	557.98 MB/s
16	64	1187	732125	46.57 MB	16 bytes	224.57 kB	156.80 GB	301.79 MB/s	733220	25.59 MB	328 bytes	400.81 kB	280.27 GB	539.44 MB/s

Note: data for OOMED lines are (obviously) only until the OOME has occurred and are thus not complete.

It's evident that the current receive predictor size of 32kb leads to an excessive increase in allocated objects outside of TLABs (which is way more expensive than allocation within a TLAB) for smaller heap sizes of 1GB and 2GB and we have very small average object sizes in regions outside of TLABs.

I have a vague theory at the moment why this happens: When a request arrives, Netty places it into a buffer and accumulates these buffers (in a CompositeByteBuf) until all data from an HTTP request have been received. Now, the PMC track is a full-text benchmark and hence bulk requests are rather large. So we allocate a lot of smaller objects which then get occasionally moved to a larger buffer (the CompositeByteBuf). The JVM sizes TLABs adaptively based on recent allocation requests and with a receive predictor size we seem to have an allocation pattern (on smaller heaps) that tricks the JVM into making TLABs too small (I did one experiment running on a 2GB heap with -XX:MinTLABSize=65536 instead of the default -XX:MinTLABSize=2048 and got much saner results). I am not yet sure why this only happens with small heap sizes but I think it is related to the size of the young generation (or to be more precise: eden) but I did not test this yet.

Next steps

I may do further analysis of TLAB sizes based on the raw data from the GC log if that is necessary but first I will continue the experiments with our other tracks and also a track with a mixed workload which also models a more real-world use case.

Unfortunately, my original testing was based only 4GB heaps where a 32kb receive predictor size works really well for all our benchmarks. Based on this results where I look at a much broader range of heap sizes I think the pragmatic choice is 64kb.