[SPARK-10532][EC2]Added --profile option to specify the name of profile

[teramonagi]

No description provided.

[shivaram]

@teramonagi Could you clarify what these profiles are meant to be ? Are they IAM profiles or something else ?

cc @nchammas

[teramonagi]

@shivaram
"profiles" give us the way that you can specify the set of credentials you want to use when you initialize a connection to AWS.

You can keep multiple sets of credentials in the same credentials files using different profile names.
For example, you can use --profile option to do that when you use "aws cli tool".

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

(Search the word "--profile" on the abobe link)

[neil-rubens]

+1

using AWS profile is considered to be best practice: https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs

[JoshRosen]

Can you expand this to explain what type of profile this is talking about? Like, maybe clarify whether it's an IAM or boto profile or whatever? As it stands now, I worry that this description isn't clear to people unfamiliar with this feature.

[teramonagi]

Actually speaking, we can use any types of profile(AWS/boto). The remaining problem is just the loading order of these "profiles" as Jan Vlcinsky explained. I think that it is little bit long to explain that in commandline help.

I adde little modification(aws or boto config) in that help.

[JoshRosen]

What if I wanted to have a profile that's named "default"? I think it would better to have None as the default and to replace this with an is not None check.

[teramonagi]

LGTM.
I will fix this quickly

[JoshRosen]

This seems reasonable to me. Might be nice to have @nchammas take a quick peek. If I don't hear otherwise, I'll merge this by Monday.

[nchammas]

LGTM. I would update the PR description though so when this gets merged in the commit message has the rationale behind this change.

[JoshRosen]

Jenkins, this is ok to test.

[JoshRosen]

I'm going to run the Python style checker, then will merge this and update the PR description myself. Thanks @teramonagi.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #44004 has finished for PR 8696 at commit 5152fa4.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[teramonagi]

@JihongMA Thanks a lot Josh!!!