upload_data

[frdchang]

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):
• Framework Version:
• Python Version: 3
• CPU or GPU: CPU
• Python SDK Version:
• Are you using a custom image: no

Describe the problem

upload_data() under sessions.py can not upload to encrypted s3 buckets right now. Fortunately, The boto3 function that upload_data() is based on, upload_file(), can accept 'ExtraArgs' as a parameter, in which the encryption key and such can be defined.

For example, upload_data may look like the following, with ExtraArgs being passed to upload_file
upload_data(....,ExtraArgs = {'ServerSideEncryption':'aws:kms', 'SSEKMSKeyId':'alias/aws/s3'})

note that estimator uses a different format to pass in the KMSKEY info. Perhaps there is a standard way of passing this info.

Minimal repro / logs

Please provide any logs and a bare minimum reproducible test case, as this will be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

• Exact command to reproduce:

[jesterhazy]

Hi @frdchang, thanks for using SageMaker!

Your request looks like a small change, I'll mark it as contributions welcome and you can submit a Pull Request.

[drajeshk]

Hi @frdchang , Is it ok if I submit a PR for this issue?

[drajeshk]

PR #836 submitted

[laurenyu]

change has been released in v1.26.0

[frdchang]

Hello, thanks for looking into this! Fred

…

On Wed, May 29, 2019 at 8:56 AM drajeshk ***@***.***> wrote: Hi @frdchang <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_frdchang&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=ntk_R0Xr6EzKP82HCwMlCrAA3g45LadJCBqf7G6RaPM&m=IZTFo85ThWa6UrY6cEAKkmNSRxA1fNWHQaa1Ntoqi80&s=1j_qMcKjEGy1Fy1EG6hMtVeWKYM0eOL0gEVsy-ejCMQ&e=> , Is it ok if I submit a PR for this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_aws_sagemaker-2Dpython-2Dsdk_issues_774-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAAYIIOWPCU6GFRV4Y27K64TPX2RSJA5CNFSM4HKWCHWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPZMLQ-23issuecomment-2D496997934&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=ntk_R0Xr6EzKP82HCwMlCrAA3g45LadJCBqf7G6RaPM&m=IZTFo85ThWa6UrY6cEAKkmNSRxA1fNWHQaa1Ntoqi80&s=sTnCfHtut9tFqK_Nd5xLrxMHGv356YV0WyXp690X8j8&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAYIIOVM5P7LATKVJ2HRHVDPX2RSJANCNFSM4HKWCHWA&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=ntk_R0Xr6EzKP82HCwMlCrAA3g45LadJCBqf7G6RaPM&m=IZTFo85ThWa6UrY6cEAKkmNSRxA1fNWHQaa1Ntoqi80&s=a-dCUewTzMer2bl7oux07UXZhMUiVaINkY780RxFATI&e=> .

[ajsilva-code]

Try this (you need to copy the S3 bucket KMS encryption key):

bucket = 'BucketName'
prefix = 'Whatever'

test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix = prefix, bucket = bucket, extra_args = {"ServerSideEncryption": "aws:kms", "SSEKMSKeyId": "xxxxxx-xxxxx-xxxxx"})