diff --git a/Makefile b/Makefile index e88271afd7..81f8564357 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ SHELL = /usr/bin/env bash -o pipefail BUILD_IMAGE ?= true DOCKER ?= docker MINIKUBE_PROFILE ?= minikube -DEPENDENCIES ?= ct helm helm-docs java21 git +DEPENDENCIES ?= ct helm helm-docs java21 git yamllint OPTIONAL_DEPENDENCIES := jq kubectl minikube VENV_DIR := .venv PYTHON_CLIENT_DIR := client/python @@ -217,7 +217,7 @@ helm-unittest: check-dependencies ## Run Helm chart unittest @helm unittest helm/polaris @echo "--- Helm chart unittest complete ---" -helm-lint: DEPENDENCIES := ct +helm-lint: DEPENDENCIES := ct yamllint .PHONY: helm-lint helm-lint: check-dependencies ## Run Helm chart lint check @echo "--- Running Helm chart linting ---" diff --git a/helm/polaris/README.md b/helm/polaris/README.md index bbfc8557df..3a7302b083 100644 --- a/helm/polaris/README.md +++ b/helm/polaris/README.md @@ -46,12 +46,14 @@ A Helm chart for Apache Polaris (incubating). The below instructions assume Minikube and Helm are installed. -Start the Minikube cluster, build and load image into the Minikube cluster: - +Start the Minikube cluster: ```bash minikube start eval $(minikube docker-env) +``` +If you have cloned Polaris git repository, you may want to build and load image into the Minikube cluster: +```bash ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ @@ -60,10 +62,24 @@ eval $(minikube docker-env) -Dquarkus.container-image.build=true ``` +Otherwise load ready images from Docker Hub: +```bash +minikube image pull apache/polaris:latest +minikube image pull apache/polaris-admin-tool:latest +``` + ### Installing the chart locally The below instructions assume a local Kubernetes cluster is running and Helm is installed. +> [!NOTE] +> The examples below assume that you have cloned Polaris git repository, which contains the Polaris Helm chart in the `helm/polaris` directory. +> If you do not want to clone the repository or want to use an official Helm chart release, you can download it from the [Apache Polaris Helm Chart repository](https://downloads.apache.org/incubator/polaris/helm-chart/): +> ```bash +> helm pull polaris --repo https://downloads.apache.org/incubator/polaris/helm-chart --version 1.2.0-incubating --untar --untardir ./helm +> ``` +> It will unpack the chart into `./helm/polaris` directory, so that scripts below can be used as is. + #### Common setup Create the target namespace: @@ -75,7 +91,11 @@ Create all the required resources in the `polaris` namespace. This usually inclu database, Kubernetes secrets, and service accounts. The Polaris chart does not create these resources automatically, as they are not required for all Polaris deployments. The chart will fail if these resources are not created beforehand. You can find some examples in the -`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. +`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. E.g. you can run the command: +```bash +kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ +kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s +``` Below are two sample deployment models for installing the chart: one with a non-persistent backend and another with a persistent backend. @@ -105,26 +125,29 @@ helm upgrade --install --namespace polaris \ kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=polaris --timeout=120s ``` -To access Polaris and Postgres locally, set up port forwarding for both services (This is needed for bootstrap processes): +To access Polaris and Postgres locally, set up port forwarding for both services (this might be needed if you want to use Polaris command line tool or if you want to bootstrap the catalog by calling Polaris admin tool locally): ```bash -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 & -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 & ``` -Run the catalog bootstrap using the Polaris admin tool. This step initializes the catalog with the required configuration: +Run the catalog bootstrap using the Polaris admin tool in minikube docker environment. This step initializes the catalog with the required configuration: ```bash -container_envs=$(kubectl exec -it -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) -export QUARKUS_DATASOURCE_USERNAME=$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_PASSWORD=$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_JDBC_URL=$(echo "$container_envs" | grep quarkus.datasource.jdbc.url | sed 's/postgres/localhost/2' | awk -F '=' '{print $2}' | tr -d '\n\r') - -java -jar runtime/admin/build/quarkus-app/quarkus-run.jar bootstrap -c POLARIS,root,pass -r POLARIS +container_envs=$(kubectl exec -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) && \ +docker run --rm \ + -e QUARKUS_DATASOURCE_USERNAME="$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_PASSWORD="$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_JDBC_URL="jdbc:postgresql://$(kubectl get svc -n polaris postgres -o jsonpath='{.spec.clusterIP}'):5432/POLARIS" \ + apache/polaris-admin-tool:latest \ + bootstrap -c POLARIS,root,pass -r POLARIS ``` ### Uninstalling ```bash +kill $(lsof -ti :8181) $(lsof -ti :5432) + helm uninstall --namespace polaris polaris kubectl delete --namespace polaris -f helm/polaris/ci/fixtures/ @@ -142,11 +165,13 @@ The following tools are required to run the tests: * [Helm Unit Test](https://github.com/helm-unittest/helm-unittest) * [Chart Testing](https://github.com/helm/chart-testing) +* [yamllint](https://github.com/adrienverge/yamllint) Quick installation instructions for these tools: ```bash helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing +brew install yamllint ``` The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory diff --git a/helm/polaris/README.md.gotmpl b/helm/polaris/README.md.gotmpl index 99feabb101..11093b79f1 100644 --- a/helm/polaris/README.md.gotmpl +++ b/helm/polaris/README.md.gotmpl @@ -48,12 +48,14 @@ weight: 675 The below instructions assume Minikube and Helm are installed. -Start the Minikube cluster, build and load image into the Minikube cluster: - +Start the Minikube cluster: ```bash minikube start eval $(minikube docker-env) +``` +If you have cloned Polaris git repository, you may want to build and load image into the Minikube cluster: +```bash ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ @@ -62,10 +64,24 @@ eval $(minikube docker-env) -Dquarkus.container-image.build=true ``` +Otherwise load ready images from Docker Hub: +```bash +minikube image pull apache/polaris:latest +minikube image pull apache/polaris-admin-tool:latest +``` + ### Installing the chart locally The below instructions assume a local Kubernetes cluster is running and Helm is installed. +> [!NOTE] +> The examples below assume that you have cloned Polaris git repository, which contains the Polaris Helm chart in the `helm/polaris` directory. +> If you do not want to clone the repository or want to use an official Helm chart release, you can download it from the [Apache Polaris Helm Chart repository](https://downloads.apache.org/incubator/polaris/helm-chart/): +> ```bash +> helm pull polaris --repo https://downloads.apache.org/incubator/polaris/helm-chart --version 1.2.0-incubating --untar --untardir ./helm +> ``` +> It will unpack the chart into `./helm/polaris` directory, so that scripts below can be used as is. + #### Common setup Create the target namespace: @@ -76,8 +92,12 @@ kubectl create namespace polaris Create all the required resources in the `polaris` namespace. This usually includes a Postgres database, Kubernetes secrets, and service accounts. The Polaris chart does not create these resources automatically, as they are not required for all Polaris deployments. The chart will -fail if these resources are not created beforehand. You can find some examples in the -`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. +fail if these resources are not created beforehand. You can find some examples in the +`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. E.g. you can run the command: +```bash +kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ +kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s +``` Below are two sample deployment models for installing the chart: one with a non-persistent backend and another with a persistent backend. @@ -107,26 +127,29 @@ helm upgrade --install --namespace polaris \ kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=polaris --timeout=120s ``` -To access Polaris and Postgres locally, set up port forwarding for both services (This is needed for bootstrap processes): +To access Polaris and Postgres locally, set up port forwarding for both services (this might be needed if you want to use Polaris command line tool or if you want to bootstrap the catalog by calling Polaris admin tool locally): ```bash -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 & -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 & ``` -Run the catalog bootstrap using the Polaris admin tool. This step initializes the catalog with the required configuration: +Run the catalog bootstrap using the Polaris admin tool in minikube docker environment. This step initializes the catalog with the required configuration: ```bash -container_envs=$(kubectl exec -it -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) -export QUARKUS_DATASOURCE_USERNAME=$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_PASSWORD=$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_JDBC_URL=$(echo "$container_envs" | grep quarkus.datasource.jdbc.url | sed 's/postgres/localhost/2' | awk -F '=' '{print $2}' | tr -d '\n\r') - -java -jar runtime/admin/build/quarkus-app/quarkus-run.jar bootstrap -c POLARIS,root,pass -r POLARIS +container_envs=$(kubectl exec -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) && \ +docker run --rm \ + -e QUARKUS_DATASOURCE_USERNAME="$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_PASSWORD="$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_JDBC_URL="jdbc:postgresql://$(kubectl get svc -n polaris postgres -o jsonpath='{.spec.clusterIP}'):5432/POLARIS" \ + apache/polaris-admin-tool:latest \ + bootstrap -c POLARIS,root,pass -r POLARIS ``` ### Uninstalling ```bash +kill $(lsof -ti :8181) $(lsof -ti :5432) + helm uninstall --namespace polaris polaris kubectl delete --namespace polaris -f helm/polaris/ci/fixtures/ @@ -144,11 +167,13 @@ The following tools are required to run the tests: * [Helm Unit Test](https://github.com/helm-unittest/helm-unittest) * [Chart Testing](https://github.com/helm/chart-testing) +* [yamllint](https://github.com/adrienverge/yamllint) Quick installation instructions for these tools: ```bash helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing +brew install yamllint ``` The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory diff --git a/site/content/in-dev/unreleased/helm.md b/site/content/in-dev/unreleased/helm.md index ef82e8e675..c53567f1fc 100644 --- a/site/content/in-dev/unreleased/helm.md +++ b/site/content/in-dev/unreleased/helm.md @@ -22,23 +22,7 @@ type: docs weight: 675 --- - - -![Version: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/Version-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0-incubating-SNAPSHOT](https://img.shields.io/badge/AppVersion-1.2.0--incubating--SNAPSHOT-informational?style=flat-square) - -A Helm chart for Apache Polaris (incubating). - -**Homepage:** - -## Source Code - -* +This page documents the Polaris Helm chart, which can be used to deploy Polaris on Kubernetes. ## Installation @@ -46,12 +30,14 @@ A Helm chart for Apache Polaris (incubating). The below instructions assume Minikube and Helm are installed. -Start the Minikube cluster, build and load image into the Minikube cluster: - +Start the Minikube cluster: ```bash minikube start eval $(minikube docker-env) +``` +If you have cloned Polaris git repository, you may want to build and load image into the Minikube cluster: +```bash ./gradlew \ :polaris-server:assemble \ :polaris-server:quarkusAppPartsBuild --rerun \ @@ -60,10 +46,25 @@ eval $(minikube docker-env) -Dquarkus.container-image.build=true ``` +Otherwise load ready images from Docker Hub: +```bash +minikube image pull apache/polaris:latest +minikube image pull apache/polaris-admin-tool:latest +``` + ### Installing the chart locally The below instructions assume a local Kubernetes cluster is running and Helm is installed. +{{< alert note >}} +The examples below assume that you have cloned Polaris git repository, which contains the Polaris Helm chart in the `helm/polaris` directory. +If you do not want to clone the repository or want to use an official Helm chart release, you can download it from the [Apache Polaris Helm Chart repository](https://downloads.apache.org/incubator/polaris/helm-chart/): +```bash +helm pull polaris --repo https://downloads.apache.org/incubator/polaris/helm-chart --version 1.2.0-incubating --untar --untardir ./helm +``` +It will unpack the chart into `./helm/polaris` directory, so that scripts below can be used as is. +{{< /alert >}} + #### Common setup Create the target namespace: @@ -75,7 +76,11 @@ Create all the required resources in the `polaris` namespace. This usually inclu database, Kubernetes secrets, and service accounts. The Polaris chart does not create these resources automatically, as they are not required for all Polaris deployments. The chart will fail if these resources are not created beforehand. You can find some examples in the -`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. +`helm/polaris/ci/fixtures` directory, but beware that these are primarily intended for tests. E.g. you can run the command: +```bash +kubectl apply --namespace polaris -f helm/polaris/ci/fixtures/ +kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=postgres --timeout=120s +``` Below are two sample deployment models for installing the chart: one with a non-persistent backend and another with a persistent backend. @@ -107,26 +112,29 @@ helm upgrade --install --namespace polaris \ kubectl wait --namespace polaris --for=condition=ready pod --selector=app.kubernetes.io/name=polaris --timeout=120s ``` -To access Polaris and Postgres locally, set up port forwarding for both services (This is needed for bootstrap processes): +To access Polaris and Postgres locally, set up port forwarding for both services (this might be needed if you want to use Polaris command line tool or if you want to bootstrap the catalog by calling Polaris admin tool locally): ```bash -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') 8181:8181 & -kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 +kubectl port-forward -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=postgres -o jsonpath='{.items[0].metadata.name}') 5432:5432 & ``` -Run the catalog bootstrap using the Polaris admin tool. This step initializes the catalog with the required configuration: +Run the catalog bootstrap using the Polaris admin tool in minikube docker environment. This step initializes the catalog with the required configuration: ```bash -container_envs=$(kubectl exec -it -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) -export QUARKUS_DATASOURCE_USERNAME=$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_PASSWORD=$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r') -export QUARKUS_DATASOURCE_JDBC_URL=$(echo "$container_envs" | grep quarkus.datasource.jdbc.url | sed 's/postgres/localhost/2' | awk -F '=' '{print $2}' | tr -d '\n\r') - -java -jar runtime/admin/build/quarkus-app/quarkus-run.jar bootstrap -c POLARIS,root,pass -r POLARIS +container_envs=$(kubectl exec -n polaris $(kubectl get pod -n polaris -l app.kubernetes.io/name=polaris -o jsonpath='{.items[0].metadata.name}') -- env) && \ +docker run --rm \ + -e QUARKUS_DATASOURCE_USERNAME="$(echo "$container_envs" | grep quarkus.datasource.username | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_PASSWORD="$(echo "$container_envs" | grep quarkus.datasource.password | awk -F '=' '{print $2}' | tr -d '\n\r')" \ + -e QUARKUS_DATASOURCE_JDBC_URL="jdbc:postgresql://$(kubectl get svc -n polaris postgres -o jsonpath='{.spec.clusterIP}'):5432/POLARIS" \ + apache/polaris-admin-tool:latest \ + bootstrap -c POLARIS,root,pass -r POLARIS ``` ### Uninstalling ```bash +kill $(lsof -ti :8181) $(lsof -ti :5432) + helm uninstall --namespace polaris polaris kubectl delete --namespace polaris -f helm/polaris/ci/fixtures/ @@ -144,11 +152,13 @@ The following tools are required to run the tests: * [Helm Unit Test](https://github.com/helm-unittest/helm-unittest) * [Chart Testing](https://github.com/helm/chart-testing) +* [yamllint](https://github.com/adrienverge/yamllint) Quick installation instructions for these tools: ```bash helm plugin install https://github.com/helm-unittest/helm-unittest.git brew install chart-testing +brew install yamllint ``` The integration tests also require some fixtures to be deployed. The `ci/fixtures` directory