HADOOP-18820: cut v1ProviderReferenced; update audit docs

Cut unused v1ProviderReferenced() method

audit doc update
- auditing is enabled again
- new fs.s3a.audit.execution.interceptors option
- how to log httpclient

Change-Id: Ib0c5b86407e059d53c469151cf3d89ede65e9116

Author: steveloughran

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java

@@ -45,30 +45,16 @@ private V2Migration() { }
 
   public static final Logger SDK_V2_UPGRADE_LOG = LoggerFactory.getLogger(SDK_V2_UPGRADE_LOG_NAME);
 
-  private static final LogExactlyOnce WARN_OF_DIRECTLY_REFERENCED_CREDENTIAL_PROVIDER =
-      new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
   private static final LogExactlyOnce WARN_OF_REQUEST_HANDLERS =
       new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
 
-  /**
-   * Notes an AWS V1 credential provider being referenced directly.
-   * @param name name of the credential provider
-   */
-  public static void v1ProviderReferenced(String name) {
-    WARN_OF_DIRECTLY_REFERENCED_CREDENTIAL_PROVIDER.debug(
-        "Directly referencing AWS SDK V1 credential provider {}. AWS SDK V1 credential "
-            + "providers will be removed once S3A is upgraded to SDK V2", name);
-  }
-
-
   /**
    * Notes use of request handlers.
    */
   public static void v1RequestHandlersUsed() {
     WARN_OF_REQUEST_HANDLERS.warn(
         "The request handler interface has changed in AWS SDK V2, use exception interceptors "
-            + "once S3A is upgraded to SDK V2");
+            + "now S3A is upgraded to SDK V2");
   }
 
 }

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/auditing.md

@@ -22,7 +22,7 @@ and inside the AWS S3 SDK, immediately before the request is executed.
 The full architecture is covered in [Auditing Architecture](auditing_architecture.html);
 this document covers its use.
 
-## Important: Auditing is disabled by default
+## Important: Auditing is currently enabled
 
 Due to a memory leak from the use of `ThreadLocal` fields, this auditing feature
 leaked memory as S3A filesystem instances were created and deleted.
@@ -32,7 +32,7 @@ See [HADOOP-18091](https://issues.apache.org/jira/browse/HADOOP-18091) _S3A audi
 
 To avoid these memory leaks, auditing was disabled by default in the hadoop 3.3.2 release.
 
-As these memory leaks have now been fixed, auditing has been re-enabled.
+As these memory leaks have now been fixed, auditing has been re-enabled in Hadoop 3.3.5+
 
 To disable it, set `fs.s3a.audit.enabled` to `false`.
 
@@ -77,7 +77,7 @@ ideally even identifying the process/job generating load.
 
 ## Using Auditing
 
-Auditing is disabled by default.
+Auditing is enabled by default.
 When auditing enabled, a Logging Auditor will annotate the S3 logs through a custom
 HTTP Referrer header in requests made to S3.
 Other auditor classes may be used instead.
@@ -88,22 +88,22 @@ Other auditor classes may be used instead.
 |--------|---------|---------------|
 | `fs.s3a.audit.enabled` | Is auditing enabled? | `true` |
 | `fs.s3a.audit.service.classname` | Auditor classname | `org.apache.hadoop.fs.s3a.audit.impl.LoggingAuditor` |
-| `fs.s3a.audit.request.handlers` | List of extra subclasses of AWS SDK RequestHandler2 to include in handler chain | `""` |
+| `fs.s3a.audit.execution.interceptors` | Implementations of AWS v2 SDK `ExecutionInterceptor` to include in handler chain | `""` |
 | `fs.s3a.audit.referrer.enabled` | Logging auditor to publish the audit information in the HTTP Referrer header | `true` |
 | `fs.s3a.audit.referrer.filter` | List of audit fields to filter | `""` |
 | `fs.s3a.audit.reject.out.of.span.operations` | Auditor to reject operations "outside of a span" | `false` |
 
 
 ### Disabling Auditing.
 
-In this release of Hadoop, auditing is disabled.
+In this release of Hadoop, auditing is enabled by default.
 
 This can be explicitly set globally or for specific buckets
 
 ```xml
 <property>
   <name>fs.s3a.audit.enabled</name>
-  <value>false</value>
+  <value>true</value>
 </property>
 ```
 
@@ -162,6 +162,23 @@ correlate access by S3 clients to the actual operations taking place.
 Note: this logging is described as "Best Effort". There's no guarantee as to
 when logs arrive.
 
+### Integration with AWS SDK request processing
+
+The auditing component inserts itself into the AWS SDK request processing
+code, so it can attach the referrer header.
+
+It is possible to declare extra classes to add to the processing chain,
+all of which must implement the interface `software.amazon.awssdk.core.interceptor.ExecutionInterceptor`.
+
+The list of classes is set in the configuration option `fs.s3a.audit.execution.interceptors`.
+
+Before the upgrade to the V2 SDK, a list of extra subclasses of the AWS SDK `com.amazonaws.handlers.RequestHandler2`
+class could be declared in the option `fs.s3a.audit.request.handlers`;
+these would be wired up into the V1 request processing pipeline.
+
+This option is now ignored completely, other than printing a warning message the first time a filesystem is created with a non-empty value.
+
+
 ### Rejecting out-of-span operations
 
 The logging auditor can be configured to raise an exception whenever
@@ -201,8 +218,8 @@ The HTTP referrer header is attached by the logging auditor.
 If the S3 Bucket is configured to log requests to another bucket, then these logs
 entries will include the audit information _as the referrer_.
 
-This can be parsed (consult AWS documentation for a regular expression)
-and the http referrer header extracted.
+The S3 Server log entries can be parsed (consult AWS documentation for a regular expression)
+and the http referrer header extracted. 
 
 ```
 https://audit.example.org/hadoop/1/op_rename/3c0d9b7e-2a63-43d9-a220-3c574d768ef3-3/
@@ -242,13 +259,14 @@ If any of the field values were `null`, the field is omitted.
 
 _Notes_
 
-* Thread IDs are from the current thread in the JVM, so can be compared to those in`````````
+* Thread IDs are from the current thread in the JVM, so can be compared to those in 
   Log4J logs. They are never unique.
 * Task Attempt/Job IDs are only ever set during operations involving the S3A committers, specifically
-  all operations excecuted by the committer.
+  all operations executed by the committer.
   Operations executed in the same thread as the committer's instantiation _may_ also report the
   IDs, even if they are unrelated to the actual task. Consider them "best effort".
 
+Thread IDs are generated as follows: 
 ```java
 Long.toString(Thread.currentThread().getId())
 ```
@@ -269,6 +287,8 @@ This is why the span ID is always passed in as part of the URL,
 rather than just an HTTP query parameter: even if
 the header is chopped, the span ID will always be present.
 
+As of August 2023, this header is not collected in AWS CloudTrail -only S3 Server logs.
+
 ## Privacy Implications of HTTP Referrer auditing
 
 When the S3A client makes requests of an S3 bucket, the auditor
@@ -423,6 +443,12 @@ log4j.logger.org.apache.hadoop.fs.s3a.audit=TRACE
 
 This is very noisy and not recommended in normal operation.
 
+If logging of HTTP IO is enabled then the "referer" header is printed as part of every request:
+```
+log4j.logger.org.apache.http=DEBUG
+log4j.logger.software.amazon.awssdk.thirdparty.org.apache.http.client.HttpClient=DEBUG
+```
+
 ## Integration with S3A Committers
 
 Work submitted through the S3A committer will have the job (query) ID associated

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md

@@ -371,3 +371,11 @@ the interface `org.apache.hadoop.fs.s3a.audit.AWSAuditEventCallbacks`
 
 Examine the interface and associated implementations to
 see how to migrate.
+
+The option `fs.s3a.audit.request.handlers` to declare a list of v1 SDK
+`com.amazonaws.handlers.RequestHandler2` implementations to include
+in the AWS request chain is no longer supported: a warning is printed
+and the value ignored.
+
+The V2 SDK equivalent, classes implementing `software.amazon.awssdk.core.interceptor.ExecutionInterceptor`
+can be declared in the configuration option `fs.s3a.audit.execution.interceptors`.