DataProtectionProviderTests.System_UsesProvidedDirectoryAndCertificate throws a CryptographicException on the Windows.10.Arm64.Open Helix queue

[halter73]

DataProtectionProviderTests.System_UsesProvidedDirectoryAndCertificate throws CryptographicException claiming "Keyset does not exist" on the Windows.10.Arm64.Open Helix queue.

System.Security.Cryptography.CryptographicException : Keyset does not exist
Stack trace
   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.GetKeyFromCert(EncryptedKey encryptedKey, KeyInfoX509Data keyInfo) in /_/src/DataProtection/DataProtection/src/XmlEncryption/EncryptedXmlDecryptor.cs:line 147
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.EncryptedXmlWithCertificateKeys.DecryptEncryptedKey(EncryptedKey encryptedKey) in /_/src/DataProtection/DataProtection/src/XmlEncryption/EncryptedXmlDecryptor.cs:line 109
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement) in /_/src/DataProtection/DataProtection/src/XmlEncryption/EncryptedXmlDecryptor.cs:line 68
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator) in /_/src/DataProtection/DataProtection/src/XmlEncryption/XmlEncryptionExtensions.cs:line 21
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement) in /_/src/DataProtection/DataProtection/src/KeyManagement/XmlKeyManager.cs:line 454
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0() in /_/src/DataProtection/DataProtection/src/KeyManagement/DeferredKey.cs:line 45
   at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.get_Value()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor() in /_/src/DataProtection/DataProtection/src/KeyManagement/KeyBase.cs:line 51
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key) in /_/src/DataProtection/DataProtection/src/AuthenticatedEncryption/CngGcmAuthenticatedEncryptorFactory.cs:line 32
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor() in /_/src/DataProtection/DataProtection/src/KeyManagement/KeyBase.cs:line 59
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.KeyHolder.GetEncryptorInstance(Boolean& isRevoked) in /_/src/DataProtection/DataProtection/src/KeyManagement/KeyRing.cs:line 82
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.get_DefaultAuthenticatedEncryptor() in /_/src/DataProtection/DataProtection/src/KeyManagement/KeyRing.cs:line 45
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext) in /_/src/DataProtection/DataProtection/src/KeyManagement/KeyRingBasedDataProtector.cs:line 102
   at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Protect(IDataProtector protector, String plaintext) in /_/src/DataProtection/Abstractions/src/DataProtectionCommonExtensions.cs:line 199
   at Microsoft.AspNetCore.DataProtection.Da

https://dev.azure.com/dnceng/public/_build/results?buildId=850263&view=ms.vss-test-web.build-test-results-tab&runId=27098036&resultId=102965&paneView=debug

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[HaoK]

@blowdart any ideas about why we might get a keyset does not exist error on arm machines? From some googling it looks like this sometimes might be due to permission issues for the process, its weird that we'd only see this on windows arm though?

[HaoK]

This looks potentially like a real product issue as opposed to a flaky test

[blowdart]

No idea. I suggest @bartonjs is the expert here

[bartonjs]

It generally means you did one of

• Import a PFX without PersistKeySet, add it to a persisted cert store, original object gets finalized and cleans up temporary key (then something using the persisted store copy blows up).
• Import a PFX, intentionally not using PersistKeySet, end up with complicated copies of the cert object, one gets cleaned up and erases the temporary private key (and the others blow up).

Alternatively, something other than .NET could have erased the key (e.g. an after-test state wipe). But usually this comes from "added to a persisted store without loading with PersistKeySet".

[HaoK]

Strange, this test has been running for a long time without issues on windows/ubuntu (including arm), is it possible something is slightly different on windows arm64 (which was added to our helix runs last year) with some of the pfx behaviors you mention?

The test code that's failing is the following, I see a X509KeyStorageFlags.Exportable, is that what I should be double checking?

        public void System_UsesProvidedDirectoryAndCertificate()
        {
            var filePath = Path.Combine(GetTestFilesPath(), "TestCert.pfx");
            using (var store = new X509Store(StoreName.My, StoreLocation.CurrentUser))
            {
                store.Open(OpenFlags.ReadWrite);
                store.Add(new X509Certificate2(filePath, "password", X509KeyStorageFlags.Exportable));
                store.Close();
            }

            WithUniqueTempDirectory(directory =>
            {
                var certificateStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);
                certificateStore.Open(OpenFlags.ReadWrite);
                var certificate = certificateStore.Certificates.Find(X509FindType.FindBySubjectName, "TestCert", false)[0];
                Assert.True(certificate.HasPrivateKey, "Cert should have a private key");
                try
                {
                    // Step 1: directory should be completely empty
                    directory.Create();
                    Assert.Empty(directory.GetFiles());

                    // Step 2: instantiate the system and round-trip a payload
                    var protector = DataProtectionProvider.Create(directory, certificate).CreateProtector("purpose");
                    var data = protector.Protect("payload");

                    // add a cert without the private key to ensure the decryption will still fallback to the cert store
                    var certWithoutKey = new X509Certificate2(Path.Combine(GetTestFilesPath(), "TestCertWithoutPrivateKey.pfx"), "password");
                    var unprotector = DataProtectionProvider.Create(directory, o => o.UnprotectKeysWithAnyCertificate(certWithoutKey)).CreateProtector("purpose");
                    Assert.Equal("payload", unprotector.Unprotect(data));

                    // Step 3: validate that there's now a single key in the directory and that it's is protected using the certificate
                    var allFiles = directory.GetFiles();
                    Assert.Single(allFiles);
                    Assert.StartsWith("key-", allFiles[0].Name, StringComparison.OrdinalIgnoreCase);
                    string fileText = File.ReadAllText(allFiles[0].FullName);
                    Assert.DoesNotContain("Warning: the key below is in an unencrypted form.", fileText, StringComparison.Ordinal);
                    Assert.Contains("X509Certificate", fileText, StringComparison.Ordinal);
                }
                finally
                {
                    certificateStore.Remove(certificate);
                    certificateStore.Close();
                }
            });

[bartonjs]

is it possible something is slightly different on windows arm64 (which was added to our helix runs last year) with some of the pfx behaviors you mention?

Probably just that there's less memory available, so finalization runs more often.

            using (var store = new X509Store(StoreName.My, StoreLocation.CurrentUser))
            {
                store.Open(OpenFlags.ReadWrite);
                store.Add(new X509Certificate2(filePath, "password", X509KeyStorageFlags.Exportable));
                store.Close();
            }

This is precisely the bug.

• A PFX is loaded without PersistKeySet
• It is added to a persisted store
• The loaded cert object is no longer rooted, during GC it gets put on the finalizer queue
• Eventually the finalizer runs, calling the "clean up your temporary file" routine
• If the test is still running it explodes.

Since you do seem to want the state back to the original you don't want to add PersistKeySet, instead you want to change your code to something like

        public void System_UsesProvidedDirectoryAndCertificate()
        {
            var filePath = Path.Combine(GetTestFilesPath(), "TestCert.pfx");
            using (X509Certificate2 imported = new X509Certificate2(filePath, "password", X509KeyStorageFlags.Exportable))
            {
                using (var store = new X509Store(StoreName.My, StoreLocation.CurrentUser))
                {
                    store.Open(OpenFlags.ReadWrite);
                    store.Add(imported);
                    store.Close();
                }

              WithUniqueTempDirectory(directory =>
              {
                  // etc, everything as it is.
              });
          }
      }

Now the imported certificate lasts for the life of the test method (so it's not eligible for finalization during test method execution), and once the test method finishes it deterministically cleans up the temporary key file.