HADOOP-19161. Docs

Change-Id: I5f8e23c8481e8085223eb862882c51e56e84703d

Author: steveloughran

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md

@@ -180,7 +180,11 @@ The S3A Filesystem client supports the notion of input policies, similar
 to that of the Posix `fadvise()` API call. This tunes the behavior of the S3A
 client to optimise HTTP GET requests for the different use cases.
 
-### fadvise `sequential`
+The list of supported options is found in
+[FSDataInputStream](../../../../../../hadoop-common-project/hadoop-common/target/site/filesystem/fsdatainputstreambuilder.html).
+
+
+### fadvise `sequential`, `whole-file`
 
 Read through the file, possibly with some short forward seeks.
 
@@ -196,6 +200,9 @@ sequential access, as should those reading data from gzipped `.gz` files.
 Because the "normal" fadvise policy starts off in sequential IO mode,
 there is rarely any need to explicit request this policy.
 
+Distcp will automatically request `whole-file` access, even on deployments
+where the cluster configuration is for `random` IO.
+
 ### fadvise `random`
 
 Optimised for random IO, specifically the Hadoop `PositionedReadable`
@@ -243,7 +250,7 @@ basis.
 to set fadvise policies on input streams. Once implemented,
 this will become the supported mechanism used for configuring the input IO policy.
 
-### fadvise `normal` (default)
+### fadvise `normal` or `adaptive` (default)
 
 The `normal` policy starts off reading a file  in `sequential` mode,
 but if the caller seeks backwards in the stream, it switches from
@@ -276,7 +283,45 @@ Fix: Use one of the dedicated [S3A Committers](committers.md).
 
 ## <a name="tuning"></a> Options to Tune
 
-### <a name="pooling"></a> Thread and connection pool settings.
+### <a name="flags"></a> Performance Flags: `fs.s3a.performance.flag`
+
+This option takes a comma separated list of performance flags.
+View it as the equivalent of the `-O` compiler optimization list C/C++ compilers offer.
+That is a complicated list of options which deliver speed if the person setting them
+understands the risks.
+
+* The list of flags MAY change across releases
+* The semantics of specific flags SHOULD NOT change across releases.
+* If an option is to be tuned which may relax semantics, a new option MUST be defined.
+* Unknown flags are ignored; this is to avoid compatibility.
+* The option `*` means "turn everything on". This is implicitly unstable across releases.
+
+| *Option* | *Meaning*          | Since |
+|----------|--------------------|:------|
+| `create` | Create Performance | 3.4.1 |
+
+The `create` flag has the same semantics as [`fs.s3a.create.performance`](#create-performance)
+
+
+### <a name="create-performance"></a> Create Performance `fs.s3a.create.performance`
+
+
+The configuration option `fs.s3a.create.performance` has the same behavior as
+the `fs.s3a.performance.flag` flag option `create`:
+
+* No overwrite checks are made when creating a file, even if overwrite is set to `false` in the application/library code
+* No checks are made for an object being written above a path containing other objects (i.e. a "directory")
+* No checks are made for a parent path containing an object which is not a directory marker (i.e. a "file")
+
+This saves multiple probes per operation, especially a `LIST` call.
+
+It may however result in
+* Unintentional overwriting of data
+* Creation of directory structures which can no longer be navigated through filesystem APIs.
+
+Use with care, and, ideally, enable versioning on the S3 store.
+
+### <a name="threads"></a> Thread and connection pool settings.
 
 Each S3A client interacting with a single bucket, as a single user, has its
 own dedicated pool of open HTTP connections alongside a pool of threads used
@@ -441,16 +486,16 @@ killer.
 1. As discussed [earlier](#pooling), use large values for
 `fs.s3a.threads.max` and `fs.s3a.connection.maximum`.
 
-1. Make sure that the bucket is using `sequential` or `normal` fadvise seek policies,
-that is, `fs.s3a.experimental.input.fadvise` is not set to `random`
-
 1. Perform listings in parallel by setting `-numListstatusThreads`
 to a higher number. Make sure that `fs.s3a.connection.maximum`
 is equal to or greater than the value used.
 
 1. If using `-delete`, set `fs.trash.interval` to 0 to avoid the deleted
 objects from being copied to a trash directory.
 
+1. If using distcp to upload to a new path where no existing data exists,
+   consider adding the option `create` to the flags in `fs.s3a.performance.flag`.
+
 *DO NOT* switch `fs.s3a.fast.upload.buffer` to buffer in memory.
 If one distcp mapper runs out of memory it will fail,
 and that runs the risk of failing the entire job.
@@ -461,12 +506,6 @@ efficient in terms of HTTP connection use, and reduce the IOP rate against
 the S3 bucket/shard.
 
 ```xml
-
-<property>
-  <name>fs.s3a.experimental.input.fadvise</name>
-  <value>normal</value>
-</property>
-
 <property>
   <name>fs.s3a.block.size</name>
   <value>128M</value>
@@ -481,6 +520,12 @@ the S3 bucket/shard.
   <name>fs.trash.interval</name>
   <value>0</value>
 </property>
+
+<!-- maybe -->
+<property>
+  <name>fs.s3a.create.performance</name>
+  <value>create</value>
+</property>
 ```
 
 ## <a name="rm"></a> hadoop shell commands `fs -rm`
@@ -719,20 +764,20 @@ exception and S3A initialization will fail.
 
 Supported values for `fs.s3a.ssl.channel.mode`:
 
-| `fs.s3a.ssl.channel.mode` Value | Description |
-|-------------------------------|-------------|
-| `default_jsse` | Uses Java JSSE without GCM on Java 8 |
-| `default_jsse_with_gcm` | Uses Java JSSE |
-| `default` | Uses OpenSSL, falls back to `default_jsse` if OpenSSL cannot be loaded |
-| `openssl` | Uses OpenSSL, fails if OpenSSL cannot be loaded |
+| `fs.s3a.ssl.channel.mode` Value | Description                                                            |
+|---------------------------------|------------------------------------------------------------------------|
+| `default_jsse`                  | Uses Java JSSE without GCM on Java 8                                   |
+| `default_jsse_with_gcm`         | Uses Java JSSE                                                         |
+| `default`                       | Uses OpenSSL, falls back to `default_jsse` if OpenSSL cannot be loaded |
+| `openssl`                       | Uses OpenSSL, fails if OpenSSL cannot be loaded                        |
 
 The naming convention is setup in order to preserve backwards compatibility
 with the ABFS support of [HADOOP-15669](https://issues.apache.org/jira/browse/HADOOP-15669).
 
 Other options may be added to `fs.s3a.ssl.channel.mode` in the future as
 further SSL optimizations are made.
 
-### WildFly classpath requirements
+### WildFly classpath and SSL library requirements
 
 For OpenSSL acceleration to work, a compatible version of the
 wildfly JAR must be on the classpath. This is not explicitly declared
@@ -742,14 +787,21 @@ optional.
 If the wildfly JAR is not found, the network acceleration will fall back
 to the JVM, always.
 
-Note: there have been compatibility problems with wildfly JARs and openSSL
+Similarly, the `libssl` library must be compatibile with wildfly.
+
+Wildfly requires this native library to be part of an `openssl` installation.
+Third party implementations may not work correctly.
+This can be an isse in FIPS-compliant deployments, where the `libssl` library
+is a third-party implementation built with restricted TLS protocols.
+
+
+There have been compatibility problems with wildfly JARs and openSSL
 releases in the past: version 1.0.4.Final is not compatible with openssl 1.1.1.
 An extra complication was older versions of the `azure-data-lake-store-sdk`
 JAR used in `hadoop-azure-datalake` contained an unshaded copy of the 1.0.4.Final
 classes, causing binding problems even when a later version was explicitly
 being placed on the classpath.
 
-
 ## <a name="initilization"></a> Tuning FileSystem Initialization.
 
 ### Bucket existence checks