Recognise direct buffers in heap size docs (#42070)

This commit slightly reworks the recommendations in the docs about setting the
heap size:

* the "rules of thumb" are actually instructions that should be followed

* the reason for setting `Xmx` to 50% of the heap size is more subtle than just
  leaving space for the filesystem cache

* it is normal to see Elasticsearch using more memory than `Xmx`

* replace `cutoff` and `limit` with `threshold` since all three terms are used
  interchangeably

* since we recommend setting `Xmx` equal to `Xms`, avoid talking about setting
  `Xmx` in isolation

Relates #41954

Author: DaveCTurner

docs/reference/setup/important-settings/heap-size.asciidoc

@@ -7,42 +7,48 @@ to ensure that Elasticsearch has enough heap available.
 
 Elasticsearch will assign the entire heap specified in
 <<jvm-options,jvm.options>> via the `Xms` (minimum heap size) and `Xmx` (maximum
-heap size) settings.
+heap size) settings. You should set these two settings to be equal to each
+other.
 
 The value for these setting depends on the amount of RAM available on your
-server. Good rules of thumb are:
+server:
 
-* Set the minimum heap size (`Xms`) and maximum heap size (`Xmx`) to be equal to
-  each other.
+* Set `Xmx` and `Xms` to no more than 50% of your physical RAM. {es} requires
+  memory for purposes other than the JVM heap and it is important to leave
+  space for this. For instance, {es} uses off-heap buffers for efficient
+  network communication, relies on the operating system's filesystem cache for
+  efficient access to files, and the JVM itself requires some memory too. It is
+  normal to observe the {es} process using more memory than the limit
+  configured with the `Xmx` setting.
 
-* The more heap available to Elasticsearch, the more memory it can use for
-  caching. But note that too much heap can subject you to long garbage
-  collection pauses.
-
-* Set `Xmx` to no more than 50% of your physical RAM, to ensure that there is
-  enough physical RAM left for kernel file system caches.
-
-* Don’t set `Xmx` to above the cutoff that the JVM uses for compressed object
-  pointers (compressed oops); the exact cutoff varies but is near 32 GB. You can
-  verify that you are under the limit by looking for a line in the logs like the
-  following:
+* Set `Xmx` and `Xms` to no more than the threshold that the JVM uses for
+  compressed object pointers (compressed oops); the exact threshold varies but
+  is near 32 GB. You can verify that you are under the threshold by looking for a
+  line in the logs like the following:
 +
     heap size [1.9gb], compressed ordinary object pointers [true]
 
-* Even better, try to stay below the threshold for zero-based compressed oops;
-  the exact cutoff varies but 26 GB is safe on most systems, but can be as large
-  as 30 GB on some systems. You can verify that you are under the limit by
-  starting Elasticsearch with the JVM options `-XX:+UnlockDiagnosticVMOptions
-  -XX:+PrintCompressedOopsMode` and looking for a line like the following:
+* Ideally set `Xmx` and `Xms` to no more than the threshold for zero-based
+  compressed oops; the exact threshold varies but 26 GB is safe on most
+  systems, but can be as large as 30 GB on some systems. You can verify that
+  you are under this threshold by starting {es} with the JVM options
+  `-XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode` and looking for
+  a line like the following:
 +
 --
     heap address: 0x000000011be00000, size: 27648 MB, zero based Compressed Oops
 
-showing that zero-based compressed oops are enabled instead of
+showing that zero-based compressed oops are enabled. If zero-based compressed
+oops are not enabled then you will see a line like the following instead:
 
     heap address: 0x0000000118400000, size: 28672 MB, Compressed Oops with base: 0x00000001183ff000
 --
 
+The more heap available to {es}, the more memory it can use for its internal
+caches, but the less memory it leaves available for the operating system to use
+for the filesystem cache. Also, larger heaps can cause longer garbage
+collection pauses.
+
 Here are examples of how to set the heap size via the jvm.options file:
 
 [source,txt]
@@ -66,7 +72,7 @@ ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch <2>
 <2> Set the minimum and maximum heap size to 4000 MB.
 
 NOTE: Configuring the heap for the <<windows-service,Windows service>> is
-different than the above. The values initially populated for the Windows service
-can be configured as above but are different after the service has been
+different than the above. The values initially populated for the Windows
+service can be configured as above but are different after the service has been
 installed. Consult the <<windows-service,Windows service documentation>> for
 additional details.