[AINFRA-1533] Adopt git-crypt in this repo #14979
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| echo "$GIT_CRYPT_ENCRYPTION_KEY" | base64 -d | git-crypt unlock - |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,7 @@ | |
| # This file is `source`'d before calling `buildkite-agent pipeline upload`, and can be used | ||
| # to set up some variables that will be interpolated in the `.yml` pipeline before uploading it. | ||
|
|
||
| # "git-crypt-unlock" branch / https://github.com/Automattic/a8c-ci-toolkit-buildkite-plugin/pull/195 | ||
| export CI_TOOLKIT="automattic/a8c-ci-toolkit#0a3f10921096cee57c18ac5667fc64c1aaad4a7d" | ||
|
Comment on lines
-6
to
+7
Contributor
Author
There was a problem hiding this comment. 🎗️ TODO: Revert back to pointing to a tag version once Automattic/a8c-ci-toolkit-buildkite-plugin#195 is merged and we have an official new version of the |
||
| export TEST_COLLECTOR="test-collector#v1.10.1" | ||
| export CLAUDE_PLUGIN="claude-summarize#v1.1.0" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,13 @@ | ||
| RELEASE-NOTES.txt merge=union | ||
|
|
||
| ######################################### | ||
| # Secrets files encrypted with git-crypt | ||
| ######################################### | ||
|
|
||
| secrets.properties filter=git-crypt diff=git-crypt | ||
| sentry.properties filter=git-crypt diff=git-crypt | ||
| google-services.json filter=git-crypt diff=git-crypt | ||
| firebase.secrets.json filter=git-crypt diff=git-crypt | ||
| google-upload-credentials.json filter=git-crypt diff=git-crypt | ||
| *.keystore filter=git-crypt diff=git-crypt | ||
| *.jks filter=git-crypt diff=git-crypt |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,7 +17,7 @@ plugins { | |
|
|
||
| fladle { | ||
| variant = "vanillaDebug" | ||
| serviceAccountCredentials = rootProject.file("firebase.secrets.json") | ||
| testTargets = [ | ||
| "notPackage com.woocommerce.android.e2e.tests.screenshot", | ||
| "notClass com.woocommerce.android.e2e.tests.ui.OrdersRealAPI", | ||
|
|
@@ -496,12 +496,13 @@ android.buildTypes.all { buildType -> | |
| } | ||
|
|
||
| // If Google services file doesn't exist, copy example file. | ||
| def googleServicesFile = file("google-services.json") | ||
| if (!googleServicesFile.exists() || isFileEncrypted(googleServicesFile)) { | ||
| tasks.copyGoogleServicesExampleFile.copy() | ||
| } | ||
|
Comment on lines
+500
to
502
Contributor
There was a problem hiding this comment. If
in the docs?
Contributor
Author
There was a problem hiding this comment. Because I updated the docs before first, then later realized I had to update the Good point then, we should indeed remove that extra step from the docs. |
||
|
|
||
| // Print warning message if example Google services file is used. | ||
| if ((googleServicesFile.text) == (file("google-services.json-example").text)) { | ||
| println("WARNING: You're using the example google-services.json file. Google login will fail.") | ||
| } | ||
| } | ||
|
|
@@ -514,6 +515,13 @@ static def loadPropertiesFromFile(inputFile) { | |
| return properties | ||
| } | ||
|
|
||
| static def isFileEncrypted(File file) { | ||
| def gitcryptHeader = [0x00, 0x47, 0x49, 0x54, 0x43, 0x52, 0x59, 0x50, 0x54] as byte[] // GITCRYPT header | ||
| def header = new byte[gitcryptHeader.length] | ||
| file.withInputStream { stream -> stream.read(header) } | ||
| return Arrays.equals(header, gitcryptHeader) | ||
| } | ||
|
|
||
| def isLeakCanaryEnabled() { | ||
| return developerProperties.get("enable_leak_canary") ?: true | ||
| } | ||
|
|
||
|
Contributor
There was a problem hiding this comment. 💭 Again thinking out loud, not sure if it's worth the trouble: what do you think of grouping encrypted / secret files in the same folder and making a convention out of this for all projects (well, kinda similar to what we had before but making it more obvious)? |
|
Contributor
There was a problem hiding this comment. 💭 I was wondering if there could be a way to be sure, in reviews, that this has in fact been encrypted or not specially given files like this are binary files. Then I've noticed all
Contributor
Author
There was a problem hiding this comment. You mean like I do here? 😛 Indeed maybe we can write a Dangermattic plugin to detect which file in the PR are encrypted and add an inline comment on the file if so as an extra information? Is that what you meant? As for manually testing locally if a file is properly encrypted before pushing a commit to the remote, one can use
Contributor
There was a problem hiding this comment.
Ha, yes, I realized that function was doing that after I posted the comment 😂
Yeah, though at the same time I find it a bit difficult to do that in a systemic way that will be useful (as we don't add secrets that often)...
👍 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,11 +13,8 @@ When creating your application, you should select "**Native client**" for the ap | |
| The "**Website URL**", "**Redirect URLs**", and "**Javascript Origins**" fields are required but not used for | ||
| the mobile apps. Just use "**[https://localhost](https://localhost)**". | ||
|
|
||
| Once you've created your application in the [applications manager][wp-com-apps], you'll need to update the `wc.oauth.app_id` and `wc.oauth.app_secret` fields in `defaults.properties` (copied from `defaults-example.properties`). See [setup instructions][setup] for more details. | ||
| Then you can compile and run the app on a device or an emulator and try to login with a WordPress.com account. Note that authenticating to WordPress.com via Google is not supported in development builds of the app, only in the official release. | ||
|
|
||
| Note that credentials created with our [WordPress.com applications manager][wp-com-apps] | ||
| allow login only and not signup. New accounts must be created using the [official app][wp-app] | ||
|
|
@@ -35,7 +32,15 @@ Read more about [OAuth2][oauth] and the [WordPress.com REST endpoint][wp-api]. | |
|
|
||
| #### `secrets.properties` | ||
|
|
||
| The `secrets.properties` file is used to store sensitive information that should not be checked into version control in clear text. | ||
| This file is encrypted (using `git-crypt`), and only developers working at Automattic have the decryption key. | ||
|
|
||
| If you are a developer working at Automattic, ensure you followed those instructions once after cloning the repo: | ||
| 1. Make sure you have `git-crypt` installed (`brew install git-crypt`) | ||
| 1. Search for "WooCommerce Android git-crypt encryption key" in our Secret Store, and copy the Base64 value in your clipboard | ||
|
Contributor
There was a problem hiding this comment. nit: I wonder, is it okay to provide a direct link to Secret Store? 🤔
Contributor
Author
There was a problem hiding this comment. Yeah I hesistated on this too. It's probably OK I guess because it's just a link that won't be accessible by anyone outside Automattic, and those links just contain
Contributor
There was a problem hiding this comment. Yes, I also had second thoughts before suggesting this, but actually our security doesn't rely on obscurity of a url. I think it's fine to share the link here, MC requires multpile security measures anyway to get anything from it. |
||
| 1. Run `pbpaste | base64 -d | git-crypt unlock -` to decrypt the encrypted files (including `secrets.properties`) | ||
|
|
||
| If you are an external contributor, provide those variables in your `defaults.properties` instead: | ||
|
|
||
| | Property | Description | | ||
| |:---------------------------|:------------| | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -62,7 +62,7 @@ GLOTPRESS_APP_STRINGS_PROJECT_URL = 'https://translate.wordpress.com/projects/wo | |
| GLOTPRESS_PLAYSTORE_METADATA_PROJECT_URL = "#{GLOTPRESS_APP_STRINGS_PROJECT_URL}/release-notes/".freeze | ||
|
|
||
| APP_PACKAGE_NAME = 'com.woocommerce.android' | ||
| GOOGLE_FIREBASE_SECRETS_PATH = File.join(PROJECT_ROOT_FOLDER, 'firebase.secrets.json') | ||
|
|
||
| # Instantiate versioning classes | ||
| VERSION_CALCULATOR = Fastlane::Wpmreleasetoolkit::Versioning::MarketingVersionCalculator.new | ||
|
|
@@ -80,7 +80,7 @@ DEFAULT_BRANCH = 'trunk' | |
| REPOSITORY_NAME = 'woocommerce-android' | ||
| GH_ORG_NAME = 'woocommerce' | ||
|
|
||
| UPLOAD_TO_PLAY_STORE_JSON_KEY = File.join(PROJECT_ROOT_FOLDER, 'google-upload-credentials.json') | ||
|
|
||
| SUPPORTED_LOCALES = [ | ||
| { glotpress: 'ar', android: 'ar', google_play: 'ar', promo_config: {} }, | ||
|
|
@@ -458,8 +458,6 @@ platform :android do | |
| UI.important("Downloading latest translations for release: #{release_version_current}") | ||
| UI.user_error!("Terminating as requested. Don't forget to run the remainder of this automation manually.") unless skip_confirm || UI.confirm('Do you want to continue?') | ||
|
|
||
| # Don't check translation coverage in CI | ||
| check_translation_progress_all unless is_ci | ||
| download_translations | ||
|
|
@@ -493,8 +491,6 @@ platform :android do | |
| UI.important("Finalizing release: #{release_version_current}") | ||
| UI.user_error!("Terminating as requested. Don't forget to run the remainder of this automation manually.") unless skip_confirm || UI.confirm('Do you want to continue?') | ||
|
|
||
| # Bump the release version and build code | ||
| UI.message 'Bumping final release version and build code...' | ||
| VERSION_FILE.write_version( | ||
|
|
@@ -1250,8 +1246,11 @@ platform :android do | |
| end | ||
|
|
||
| def firebase_secret(name:) | ||
| begin | ||
| key_file_secrets = JSON.parse(File.read(GOOGLE_FIREBASE_SECRETS_PATH)) | ||
| rescue StandardError | ||
| UI.user_error!('Unable to read Firebase Secrets File – did you run `echo "…encryption-key…" | base64 -d | git-crypt unlock -` on the repo?') | ||
| end | ||
| UI.user_error!("Unable to find key `#{name}` in #{GOOGLE_FIREBASE_SECRETS_PATH}") if key_file_secrets[name].nil? | ||
| key_file_secrets[name] | ||
| end | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,7 +23,7 @@ ext { | |
|
|
||
| gradle.ext { | ||
| isCi = System.getenv('CI')?.toBoolean() ?: false | ||
| secretsPath = "secrets.properties" | ||
| secretProperties = loadPropertiesWithFallback( | ||
| logger, | ||
| file("${rootDir}/defaults.properties"), | ||
|
|
@@ -40,7 +40,12 @@ static def loadPropertiesWithFallback(Logger logger, File fallbackFile, File pri | |
| def defaultProperties = readPropertiesFromFile(fallbackFile) | ||
| def primaryProperties | ||
| if (primaryFile.exists()) { | ||
| try { | ||
| primaryProperties = readPropertiesFromFile(primaryFile) | ||
| } catch (Exception e) { | ||
| logger.warn("Failed to parse primary properties file: ${primaryFile}. Using fallback: ${fallbackFile}. Error: ${e}") | ||
| primaryProperties = new Properties() | ||
| } | ||
| } else { | ||
| logger.warn("Primary properties file not found: ${primaryFile}. Using fallback: ${fallbackFile}.") | ||
| primaryProperties = new Properties() | ||
|
|
@@ -93,8 +98,14 @@ def checkForRemoteBuildCacheOptimizedExperience() { | |
| } | ||
|
|
||
| def assertSecretsApplied() { | ||
| def secretsFile = file(gradle.ext.secretsPath) | ||
| if (!secretsFile.exists()) { | ||
| throw new GradleException("The build requested remote build cache, but the `secrets.properties` file is not found. See README.md for instructions on how to set those up.") | ||
| } | ||
| try { | ||
| readPropertiesFromFile(secretsFile) | ||
| } catch (Exception ex) { | ||
| throw new GradleException("The `secrets.properties` file exists but could not be read as a valid `.properties` file. This likely means it is still encrypted (git-crypt locked). Please decrypt it before building. See README.md for instructions.", ex) | ||
| } | ||
| } | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having the repo
git-crypt-unlocked during the branch dance that is done internally bycomment_with_manifest_diffcausedYour local changes to the following files would be overwritten by checkoutissues, especially if thegit-crypt'd files listed in.gitattributeson the HEAD branch are not the same as the ones in the BASE branch1Since we don't need any secret in practice to generate the manifest and call
process{variant}Manifest, the solution is simple: just don't bother unlocking the repo's secrets for that task.A better long-term solution to make
comment_with_manifest_diffmore resilient to situations like this would be to make it usegit worktreeinstead of switching branches in-place:git worktree add $TMP_DIR_FOR_BASE $BASE_BRANCH && cd $TMP_DIR_FOR_BASEthen run./gradlew process{variant}Manifesttherecd $CHECKOUT_DIR && rm $TMP_DIR_FOR_BASE && git worktree prunethen run./gradlew process{variant}ManifestthereThat way each checkout is done in independent folders, eliminating the risk of conflicts during the branch dance.
Footnotes
like will be the case during that transition to
git-crypt, or when we'll add a new secret file, especially if that secret file previously existed unencrypted in the BASE branch as an example file for external contributors I think? ↩