Client side encryption

[NicolasTr]

Here's some code to encrypt client-side the snapshots stored in S3 .

It uses the envelope encryption of the AWS ADK for Java (AES 256 in CBC mode). You case use keys of 128, 192 or 256 bits but you _have to_ install the JCE because the envelope encryption _will_ use 256bits keys.

The key can be specified when configuring the repository.

Example with a 128 bits AES symmetric key:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
    "type": "s3",
    "settings": {
        "bucket": "my_bucket_name",
        "region": "us-west",
        "client_side_encryption_key": {
            "symmetric": "Hk/RABbqRnhgSAMXiVUV0w=="
        }
    }
}'

Example with a 512 bits RSA key pair:

$ curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
    "type": "s3",
    "settings": {
        "bucket": "my_bucket_name",
        "region": "us-west",
        "client_side_encryption_key": {
            "public": "MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALb4IRgaDtmwsz+kC/bkKLno5bRWMDbRDztcM/N/VJIg+HQKA8ees6uznEOGe6cOZp+QHYrpTIXs36QB9vfsVdUCAwEAAQ==",
            "private": "MIIBUwIBADANBgkqhkiG9w0BAQEFAASCAT0wggE5AgEAAkEAtvghGBoO2bCzP6QL9uQouejltFYwNtEPO1wz839UkiD4dAoDx56zq7OcQ4Z7pw5mn5AdiulMhezfpAH29+xV1QIDAQABAkBEy/OVnmarD7e2XDZrdMqjbKDCOA4U7nKtvTODgQMJll0b7wA52bY/kG8gA8Q2aqiKVHDRl/EQ33bELr2A56NBAiEA4+7qC/TMw2V2q+FuW5Az8mzfCH6mNmyMLc/XwuZ2/V0CIQDNf9NRbZ0EZQCiq6iBjgYzkdJlv1tDt9erqVj0w0K62QIgZDE5IFhTSfDn4VYOtKEGtKG2yH0jgvjkBZ8/MKUt2OECIFSzDuJND560EqL5paZgZ2XyAIo3aOJsb9QtJKEdqe9hAiBgithEZGGQNV/pXweOtku/CezLvY2FJaSPShfeQfXjcg==",
        }
    }
}'

I added a test case to make sure that the files are encrypted and I modified testSimpleWorkflow() to make sure that by default the files are not encrypted.

I haven't updated the documentation yet, I would like to get some reviews first.

• Encryption
• Decryption
• Test case
• Documentation

---

• Sign the CLA

[NicolasTr]

When I'll have time, I'll re-implement it using the Client-Side Data Encryption of the AWS SDK.

[saahn]

Is there any chance for this feature to be merged soon? We would really like to enable client side encryption. :)

[NicolasTr]

Thanks for your interest. I normally should be resuming my work on this next week.

Like I said above, I'll use the client-side encryption offered by the AWS SDK for Java. There will be less code to maintain in this repository and it will be possible to easily rotate the keys.

Client-Side Data Encryption of the AWS SDK

How the Encryption Works

All encryption/decryption happens _exclusively in your application_ using a process called "envelope encryption." Your private encryption keys and your unencrypted data are never sent to AWS, so it's very important that you safely manage your encryption keys. If you lose your encryption keys, you won't be able to unencrypt your data, and you can't recover your encryption keys from AWS, since AWS doesn't know anything about them.

The goal of envelope encryption is to combine the performance of fast symmetric encryption while maintaining the secure key management that asymmetric keys provide. A one-time-use symmetric key (the envelope symmetric key) is generated by the Amazon S3 encryption client to encrypt your data, then that key is encrypted by your master key and stored alongside your data in Amazon S3. When accessing your data with the Amazon S3 encryption client, the encrypted symmetric key is retrieved and decrypted with your real key, then the data is decrypted. Another benefit of envelope encryption is that if your master key is compromised, you have the option of just re-encrypting the stored envelope symmetric keys, instead of re-encrypting all the data in your account.

Encryption

• Generate a one time use envelope symmetric key.
• Encrypt the file data using this envelope key.
• Encrypt that envelope key using a master public key or symmetric key.
• Store this encrypted envelope key with the encrypted file.
• Store a description of the master key alongside the envelope key to uniquely identify the key used to encrypt the envelope key.

Decryption

• Retrieve the encrypted envelope key you stored with the encrypted file.
• Retrieve the description of the original master key.
• If the description of the master key on hand does not match the description of the original master key, use the unique description to fetch the original master symmetric key or private key.
• Decrypt the envelope key using the master key.
• Decrypt the file data using the envelope key.

[saahn]

Thanks for the update. Looking forward to this feature!

[NicolasTr]

The encryption is now done by the AWS SDK for Java.

Public/private keys are not supported at the moment (only symmetric keys).

[saahn]

Great, thanks for the speedy work!
Do you know when this will be ready to consume? We'd love to collaborate with you on testing this, if that would be helpful :)

[NicolasTr]

I switched to base 64 encoding of the key as it will be more compact.

@saahn If you want to test, use the following zip archive (obsolete link, see below) to install the plugin. It's the version 2.3.0 of this plugin + the client-side encryption. If you need another version please let me know.

[NicolasTr]

• README.md updated
• Support for public/private key pairs added

You can test with this archive.

I think the pull request can be merged. I just need my employer to sign the CLA so I can sign it too. This should be done soon.

[NicolasTr]

CLA signed.

[dadoonet]

I guess it should be private here, right?

[NicolasTr]

Fixed.

[dadoonet]

I did a first review pass but I'd love that @imotov looks at it as well.

[dadoonet]

I think you can use assertThat(e, instanceOf(MalformedJsonException.class)); here

[NicolasTr]

Fixed.

[dadoonet]

Left a small comment. The change looks good to me though I did not test it.
@imotov WDYT?

[NicolasTr]

I tested the commits on top of the v2.3.0 tag. Only on org.elasticsearch.repositories.s3.

[dadoonet]

@NicolasTr Could you please rebase your changes on master and squash all your commits? Recent patch added by @tlrx fixes compilation issues with elasticsearch 1.4 and above.

If you can't do it quickly, could you give an estimate for this? So we can decide in which version we can support this feature.

Thanks!

[NicolasTr]

Done.

I saw that @tlrx added the support of multipart uploads in 4bc4ea6. It should work with client-side encryption (see the client-side encryption feature matrix). However:

• AES is a block cipher so chunk_size has to be a multiple of the AES block size, 16. This is already checked in the AWS SDK however it's probably better to throw an exception as soon as the error can be detected so I added this.
• CBC uses the previous ciphertext block to encrypt the current block. Therefore, if you want to use multipart upload with client-side encryption without encrypting the file twice, you have to upload the parts in order. I think it's already the case with S3OutputStream. @tlrx, could you confirm?

[tlrx]

@NicolasTr Thanks for this PR and yes, parts are uploaded in order. However when need to test this.

Again - and sorry - can you rebase your PR please?

[NicolasTr]

@tlrx Thanks for your response. I fixed the conflicts.

[NicolasTr]

I just rebased against master again.

[dadoonet]

@NicolasTr Did you sign the CLA?

[NicolasTr]

Yes, I did :)

[jondb]

@NicolasTr, @dadoonet -- very excited to see this. any idea when it will be merged?

[briw]

@NicolasTr , @dadoonet , also very excited to see this and would very much like to see it merged...

[NicolasTr]

@dadoonet I rebased on top of master. I haven't rerun the tests. What's the issue with the CLA?

[dadoonet]

Probably something went wrong on our side.
Don't worry, I'll do a manual check.

[dpbus]

Any update on this? Really excited to start taking encrypted snapshots to S3.

[jyjohnson]

+1 on using this. Any eta?

Thanks!

[dadoonet]

Hi @NicolasTr

I'm super sorry that your PR never got merged in. I'm not sure how it went out of our radar.

Well, you might know that we moved this project to elasticsearch repo.

Do you think it could be possible for you to create a new PR in elasticsearch repo? We will make sure it gets attention it deserves.

Again, sorry for this loooooong delay!

Best

[dadoonet]

@NicolasTr Do you think you could come up with a new PR on elasticsearch repository?

[NicolasTr]

@dadoonet I'm sorry I missed your previous comment. Yes, I'll do it. Let's move the discussion to the elasticsearch repo.