Commit 44f8766
committed
### What changes were proposed in this pull request?
This PR aims to redact `awsAccessKeyId` by including `accesskey` pattern.
- **Apache Spark 4.0.0-preview1**
There is no point to redact `fs.s3a.access.key` because the same value is exposed via `fs.s3.awsAccessKeyId` like the following. We need to redact all.
```
$ AWS_ACCESS_KEY_ID=A AWS_SECRET_ACCESS_KEY=B bin/spark-shell
```

### Why are the changes needed?
Since Apache Spark 1.1.0, `AWS_ACCESS_KEY_ID` is propagated like the following. However, Apache Spark does not redact them all consistently.
- #450
https://github.com/apache/spark/blob/5d16c3134c442a5546251fd7c42b1da9fdf3969e/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L481-L486
### Does this PR introduce _any_ user-facing change?
Users may see more redactions on configurations whose name contains `accesskey` case-insensitively. However, those configurations are highly likely to be related to the credentials.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47392 from dongjoon-hyun/SPARK-48930.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 1e17c39)
Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 443825a commit 44f8766
File tree
2 files changed
+2
-1
lines changed- core/src
- main/scala/org/apache/spark/internal/config
- test/scala/org/apache/spark/util
2 files changed
+2
-1
lines changedLines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1155 | 1155 | | |
1156 | 1156 | | |
1157 | 1157 | | |
1158 | | - | |
| 1158 | + | |
1159 | 1159 | | |
1160 | 1160 | | |
1161 | 1161 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1093 | 1093 | | |
1094 | 1094 | | |
1095 | 1095 | | |
| 1096 | + | |
1096 | 1097 | | |
1097 | 1098 | | |
1098 | 1099 | | |
| |||
0 commit comments