[SPARK-18535][SPARK-19720][CORE][BACKPORT-2.1] Redact sensitive information #18802
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and UI ## What changes were proposed in this pull request? This patch adds a new property called `spark.secret.redactionPattern` that allows users to specify a scala regex to decide which Spark configuration properties and environment variables in driver and executor environments contain sensitive information. When this regex matches the property or environment variable name, its value is redacted from the environment UI and various logs like YARN and event logs. This change uses this property to redact information from event logs and YARN logs. It also, updates the UI code to adhere to this property instead of hardcoding the logic to decipher which properties are sensitive. Here's an image of the UI post-redaction:  Here's the text in the YARN logs, post-redaction: ``HADOOP_CREDSTORE_PASSWORD -> *********(redacted)`` Here's the text in the event logs, post-redaction: ``...,"spark.executorEnv.HADOOP_CREDSTORE_PASSWORD":"*********(redacted)","spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD":"*********(redacted)",...`` ## How was this patch tested? 1. Unit tests are added to ensure that redaction works. 2. A YARN job reading data off of S3 with confidential information (hadoop credential provider password) being provided in the environment variables of driver and executor. And, afterwards, logs were grepped to make sure that no mention of secret password was present. It was also ensure that the job was able to read the data off of S3 correctly, thereby ensuring that the sensitive information was being trickled down to the right places to read the data. 3. The event logs were checked to make sure no mention of secret password was present. 4. UI environment tab was checked to make sure there was no secret information being displayed. Author: Mark Grover <[email protected]> Closes apache#15971 from markgrover/master_redaction.
…sole ## What changes were proposed in this pull request? This change redacts senstive information (based on `spark.redaction.regex` property) from the Spark Submit console logs. Such sensitive information is already being redacted from event logs and yarn logs, etc. ## How was this patch tested? Testing was done manually to make sure that the console logs were not printing any sensitive information. Here's some output from the console: ``` Spark properties used, including those specified through --conf and those from the properties file /etc/spark2/conf/spark-defaults.conf: (spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD,*********(redacted)) (spark.authenticate,false) (spark.executorEnv.HADOOP_CREDSTORE_PASSWORD,*********(redacted)) ``` ``` System properties: (spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD,*********(redacted)) (spark.authenticate,false) (spark.executorEnv.HADOOP_CREDSTORE_PASSWORD,*********(redacted)) ``` There is a risk if new print statements were added to the console down the road, sensitive information may still get leaked, since there is no test that asserts on the console log output. I considered it out of the scope of this JIRA to write an integration test to make sure new leaks don't happen in the future. Running unit tests to make sure nothing else is broken by this change. Author: Mark Grover <[email protected]> Closes apache#17047 from markgrover/master_redaction.
Contributor
|
ok to test |
|
Test build #80137 has started for PR 18802 at commit |
|
Test build #80171 has finished for PR 18802 at commit
|
dmvieira
force-pushed
the
feature-redact
branch
from
0096ad9 to
0ca71a8
Compare
|
Test build #80173 has started for PR 18802 at commit |
dmvieira
force-pushed
the
feature-redact
branch
from
0ca71a8 to
4424b05
Compare
|
Test build #80174 has finished for PR 18802 at commit
|
dmvieira
force-pushed
the
feature-redact
branch
from
4424b05 to
e92e24a
Compare
|
Test build #80175 has finished for PR 18802 at commit
|
dmvieira
force-pushed
the
feature-redact
branch
from
e92e24a to
49941f7
Compare
|
Test build #80176 has finished for PR 18802 at commit
|
dmvieira
force-pushed
the
feature-redact
branch
from
49941f7 to
ad23355
Compare
|
Test build #80177 has started for PR 18802 at commit |
dmvieira
force-pushed
the
feature-redact
branch
from
ad23355 to
81dc26b
Compare
|
Test build #80207 has finished for PR 18802 at commit
|
Member
|
cc @markgrover @vanzin Could you please take a look at this? |
vanzin
reviewed
Aug 7, 2017
dev/run-tests.py
Outdated
Contributor
There was a problem hiding this comment.
I see you added these changes to fix the tests, but they're unrelated to the patches you're backporting.
They should, at the very least, be a separate PR, if they're really needed.
Author
There was a problem hiding this comment.
I can remove it, but tests will fail at Jenkins
Contributor
There was a problem hiding this comment.
Then, as I suggested, open a separate PR with the fix for the tests.
dmvieira
force-pushed
the
feature-redact
branch
from
81dc26b to
7b419b4
Compare
Author
|
I removed test fixes and add another PR: #18873 from this branch |
|
Test build #80357 has finished for PR 18802 at commit
|
Contributor
|
Merging to 2.1. |
asfgit
pushed a commit
that referenced
this pull request
Aug 7, 2017
…mation ## What changes were proposed in this pull request? Backporting SPARK-18535 and SPARK-19720 to spark 2.1 It's a backport PR that redacts senstive information by configuration to Spark UI and Spark Submit console logs. Using reference from Mark Grover markapache.org PRs ## How was this patch tested? Same tests from PR applied Author: Mark Grover <[email protected]> Closes #18802 from dmvieira/feature-redact.
Contributor
|
@dmvieira please close the PR since github doesn't do it automatically. |
Author
|
Thank you @vanzin |
Member
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Backporting SPARK-18535 and SPARK-19720 to spark 2.1
It's a backport PR that redacts senstive information by configuration to Spark UI and Spark Submit console logs.
Using reference from Mark Grover [email protected] PRs
How was this patch tested?
Same tests from PR applied