diff --git a/helm/polaris/README.md b/helm/polaris/README.md index e6707091ff..74b3e55adf 100644 --- a/helm/polaris/README.md +++ b/helm/polaris/README.md @@ -46,33 +46,27 @@ A Helm chart for Apache Polaris (incubating). ## Installation -### Optional +### Prerequisites -When using a custom `persistence.xml`, a Kubernetes Secret must be created for it. Below is a sample command: -```bash -kubectl create secret generic polaris-secret -n polaris --from-file=persistence.xml -``` - -### Running the unit tests - -Helm unit tests do not require a Kubernetes cluster. To run the unit tests from the Polaris repo -root: +When using the (deprecated) EclipseLink-backed metastore, a custom `persistence.xml` is required, +and a Kubernetes Secret must be created for it. Below is a sample command: ```bash -helm unittest helm/polaris +kubectl create secret generic polaris-secret -n polaris --from-file=persistence.xml ``` ### Running locally with a Kind cluster The below instructions assume Kind and Helm are installed. -Simply run the `run.sh` script from the Polaris repo root, making sure to specify the -`--eclipse-link-deps` option: +Simply run the `run.sh` script from the Polaris repo root: ```bash ./run.sh ``` +If using the EclipseLink-backed metastore, make sure to specify the `--eclipse-link-deps` option. + This script will create a Kind cluster, deploy a local Docker registry, build the Polaris Docker images with support for Postgres and load them into the Kind cluster. (It will also create an example Deployment and Service with in-memory storage.) @@ -100,27 +94,24 @@ The below instructions assume a local Kubernetes cluster is running and Helm is #### Common setup -Create and populate the target namespace: +Create the target namespace: ```bash kubectl create namespace polaris -kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ - -kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s ``` -The `helm/polaris/ci` contains a number of values files that can be used to install the chart with -different configurations. - -You can also run `ct` (chart-testing): - -```bash -ct lint --charts helm/polaris -ct install --namespace polaris --debug --charts ./helm/polaris -``` +Create all the required resources in the `polaris` namespace. This usually includes a Postgres +database and a Kubernetes Secret for the `persistence.xml` file. The Polaris chart does not create +these resources automatically, as they are not required for all Polaris deployments. The chart will +fail if these resources are not created beforehand. Below are two sample deployment models for installing the chart: one with a non-persistent backend and another with a persistent backend. +> [!WARNING] +> The examples below use values files located in the `helm/polaris/ci` directory. +> **These files are intended for testing purposes primarily, and may not be suitable for production use**. +> For production deployments, create your own values files based on the provided examples. + #### Non-persistent backend Install the chart with a non-persistent backend. From Polaris repo root: @@ -131,6 +122,16 @@ helm upgrade --install --namespace polaris \ polaris helm/polaris ``` +Note: if you are running the tests on a Kind cluster started with the `run.sh` command explained +above, then you need to run `helm upgrade` as follows: + +```bash +helm upgrade --install --namespace polaris \ + --debug --values helm/polaris/ci/simple-values.yaml \ + --set=image.repository=localhost:5001/apache/polaris \ + polaris helm/polaris +``` + #### Persistent backend > [!WARNING] @@ -184,6 +185,69 @@ kubectl delete --namespace polaris -f helm/polaris/ci/fixtures/ kubectl delete namespace polaris ``` +## Development & Testing + +This section is intended for developers who want to run the Polaris Helm chart tests. + +### Prerequisites + +The following tools are required to run the tests: + +* [Helm Unit Test](https://github.com/helm-unittest/helm-unittest) +* [Chart Testing](https://github.com/helm/chart-testing) + +Quick installation instructions for these tools: + +```bash +helm plugin install https://github.com/helm-unittest/helm-unittest.git +brew install chart-testing +``` + +The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory +contains the required resources. To deploy them, run the following command: + +```bash +kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ +kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s +``` + +The `helm/polaris/ci` contains a number of values files that will be used to install the chart with +different configurations. + +### Running the unit tests + +Helm unit tests do not require a Kubernetes cluster. To run the unit tests, execute Helm Unit from +the Polaris repo root: + +```bash +helm unittest helm/polaris +``` + +You can also lint the chart using the Chart Testing tool, with the following command: + +```bash +ct lint --charts helm/polaris +``` + +### Running the integration tests + +Integration tests require a Kubernetes cluster. See installation instructions above for setting up +a local cluster. + +Integration tests are run with the Chart Testing tool: + +```bash +ct install --namespace polaris --debug --charts ./helm/polaris +``` + +Note: if you are running the tests on a Kind cluster started with the `run.sh` command explained +above, then you need to run `ct install` as follows: + +```bash +ct install --namespace polaris --debug --charts ./helm/polaris \ + --helm-extra-set-args "--set=image.repository=localhost:5001/apache/polaris" +``` + ## Values | Key | Type | Default | Description | @@ -219,8 +283,7 @@ kubectl delete namespace polaris | extraServices | list | `[]` | Additional service definitions. All service definitions always select all Polaris pods. Use this if you need to expose specific ports with different configurations, e.g. expose polaris-http with an alternate LoadBalancer service instead of ClusterIP. | | extraVolumeMounts | list | `[]` | Extra volume mounts to add to the polaris container. See https://kubernetes.io/docs/concepts/storage/volumes/. | | extraVolumes | list | `[]` | Extra volumes to add to the polaris pod. See https://kubernetes.io/docs/concepts/storage/volumes/. | -| features | object | `{"defaults":{},"realmOverrides":{}}` | Polaris features configuration. | -| features.defaults | object | `{}` | Features to enable or disable globally. If a feature is not present in the map, the default built-in value is used. | +| features | object | `{"realmOverrides":{}}` | Polaris features configuration. | | features.realmOverrides | object | `{}` | Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used. | | fileIo | object | `{"type":"default"}` | Polaris FileIO configuration. | | fileIo.type | string | `"default"` | The type of file IO to use. Two built-in types are supported: default and wasb. The wasb one translates WASB paths to ABFS ones. | @@ -285,7 +348,7 @@ kubectl delete namespace polaris | persistence.eclipseLink.secret | object | `{"key":"persistence.xml","name":null}` | The secret name to pull persistence.xml from. | | persistence.eclipseLink.secret.key | string | `"persistence.xml"` | The key in the secret to pull persistence.xml from. | | persistence.eclipseLink.secret.name | string | `nil` | The name of the secret to pull persistence.xml from. If not provided, the default built-in persistence.xml will be used. This is probably not what you want. | -| persistence.type | string | `"relational-jdbc"` | Three built-in types are available: "relational-jdbc", "in-memory", "eclipse-link". The in-memory type is not recommended for production use. The eclipse-link type is deprecated and will be unsupported in a future release. | +| persistence.type | string | `"eclipse-link"` | The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link. | | podAnnotations | object | `{}` | Annotations to apply to polaris pods. | | podLabels | object | `{}` | Additional Labels to apply to polaris pods. | | podSecurityContext | object | `{"fsGroup":10001,"seccompProfile":{"type":"RuntimeDefault"}}` | Security context for the polaris pod. See https://kubernetes.io/docs/tasks/configure-pod-container/security-context/. | diff --git a/helm/polaris/README.md.gotmpl b/helm/polaris/README.md.gotmpl index d478c6518f..94a233ada7 100644 --- a/helm/polaris/README.md.gotmpl +++ b/helm/polaris/README.md.gotmpl @@ -47,33 +47,27 @@ ## Installation -### Optional +### Prerequisites -When using a custom `persistence.xml`, a Kubernetes Secret must be created for it. Below is a sample command: -```bash -kubectl create secret generic polaris-secret -n polaris --from-file=persistence.xml -``` - -### Running the unit tests - -Helm unit tests do not require a Kubernetes cluster. To run the unit tests from the Polaris repo -root: +When using the (deprecated) EclipseLink-backed metastore, a custom `persistence.xml` is required, +and a Kubernetes Secret must be created for it. Below is a sample command: ```bash -helm unittest helm/polaris +kubectl create secret generic polaris-secret -n polaris --from-file=persistence.xml ``` ### Running locally with a Kind cluster The below instructions assume Kind and Helm are installed. -Simply run the `run.sh` script from the Polaris repo root, making sure to specify the -`--eclipse-link-deps` option: +Simply run the `run.sh` script from the Polaris repo root: ```bash ./run.sh ``` +If using the EclipseLink-backed metastore, make sure to specify the `--eclipse-link-deps` option. + This script will create a Kind cluster, deploy a local Docker registry, build the Polaris Docker images with support for Postgres and load them into the Kind cluster. (It will also create an example Deployment and Service with in-memory storage.) @@ -101,27 +95,24 @@ The below instructions assume a local Kubernetes cluster is running and Helm is #### Common setup -Create and populate the target namespace: +Create the target namespace: ```bash kubectl create namespace polaris -kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ - -kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s ``` -The `helm/polaris/ci` contains a number of values files that can be used to install the chart with -different configurations. - -You can also run `ct` (chart-testing): - -```bash -ct lint --charts helm/polaris -ct install --namespace polaris --debug --charts ./helm/polaris -``` +Create all the required resources in the `polaris` namespace. This usually includes a Postgres +database and a Kubernetes Secret for the `persistence.xml` file. The Polaris chart does not create +these resources automatically, as they are not required for all Polaris deployments. The chart will +fail if these resources are not created beforehand. Below are two sample deployment models for installing the chart: one with a non-persistent backend and another with a persistent backend. +> [!WARNING] +> The examples below use values files located in the `helm/polaris/ci` directory. +> **These files are intended for testing purposes primarily, and may not be suitable for production use**. +> For production deployments, create your own values files based on the provided examples. + #### Non-persistent backend Install the chart with a non-persistent backend. From Polaris repo root: @@ -132,6 +123,16 @@ helm upgrade --install --namespace polaris \ polaris helm/polaris ``` +Note: if you are running the tests on a Kind cluster started with the `run.sh` command explained +above, then you need to run `helm upgrade` as follows: + +```bash +helm upgrade --install --namespace polaris \ + --debug --values helm/polaris/ci/simple-values.yaml \ + --set=image.repository=localhost:5001/apache/polaris \ + polaris helm/polaris +``` + #### Persistent backend > [!WARNING] @@ -185,4 +186,67 @@ kubectl delete --namespace polaris -f helm/polaris/ci/fixtures/ kubectl delete namespace polaris ``` +## Development & Testing + +This section is intended for developers who want to run the Polaris Helm chart tests. + +### Prerequisites + +The following tools are required to run the tests: + +* [Helm Unit Test](https://github.com/helm-unittest/helm-unittest) +* [Chart Testing](https://github.com/helm/chart-testing) + +Quick installation instructions for these tools: + +```bash +helm plugin install https://github.com/helm-unittest/helm-unittest.git +brew install chart-testing +``` + +The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory +contains the required resources. To deploy them, run the following command: + +```bash +kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ +kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s +``` + +The `helm/polaris/ci` contains a number of values files that will be used to install the chart with +different configurations. + +### Running the unit tests + +Helm unit tests do not require a Kubernetes cluster. To run the unit tests, execute Helm Unit from +the Polaris repo root: + +```bash +helm unittest helm/polaris +``` + +You can also lint the chart using the Chart Testing tool, with the following command: + +```bash +ct lint --charts helm/polaris +``` + +### Running the integration tests + +Integration tests require a Kubernetes cluster. See installation instructions above for setting up +a local cluster. + +Integration tests are run with the Chart Testing tool: + +```bash +ct install --namespace polaris --debug --charts ./helm/polaris +``` + +Note: if you are running the tests on a Kind cluster started with the `run.sh` command explained +above, then you need to run `ct install` as follows: + +```bash +ct install --namespace polaris --debug --charts ./helm/polaris \ + --helm-extra-set-args "--set=image.repository=localhost:5001/apache/polaris" +``` + {{ template "chart.valuesSection" . }} diff --git a/helm/polaris/ci/features-values.yaml b/helm/polaris/ci/features-values.yaml new file mode 100644 index 0000000000..cf44ce0c07 --- /dev/null +++ b/helm/polaris/ci/features-values.yaml @@ -0,0 +1,31 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +image: + pullPolicy: Never + +features: + ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false + SUPPORTED_CATALOG_STORAGE_TYPES: + - S3 + realmOverrides: + POLARIS: + ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: true + SUPPORTED_CATALOG_STORAGE_TYPES: + - GCS diff --git a/helm/polaris/templates/configmap.yaml b/helm/polaris/templates/configmap.yaml index ae732760cb..6d12fbd7e1 100644 --- a/helm/polaris/templates/configmap.yaml +++ b/helm/polaris/templates/configmap.yaml @@ -37,8 +37,10 @@ data: {{- /* Features */ -}} {{- range $k, $v := .Values.features -}} + {{- if ( ne $k "realmOverrides" ) -}} {{- $_ = set $map (printf "polaris.features.\"%s\"" $k) (toJson $v) -}} {{- end -}} + {{- end -}} {{- range $realm, $overrides := .Values.features.realmOverrides -}} {{- range $k, $v := $overrides -}} {{- $_ = set $map (printf "polaris.features.realm-overrides.\"%s\".\"%s\"" $realm $k) (toJson $v) -}} diff --git a/helm/polaris/values.yaml b/helm/polaris/values.yaml index 04a1013cf1..5cf964f20c 100644 --- a/helm/polaris/values.yaml +++ b/helm/polaris/values.yaml @@ -507,13 +507,12 @@ realmContext: features: # -- Features to enable or disable globally. If a feature is not present in the map, the default # built-in value is used. - defaults: {} - # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false - # SUPPORTED_CATALOG_STORAGE_TYPES: - # - S3 - # - GCS - # - AZURE - # - FILE + # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false + # SUPPORTED_CATALOG_STORAGE_TYPES: + # - S3 + # - GCS + # - AZURE + # - FILE # -- Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of # feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used. realmOverrides: {}