diff --git a/site/content/in-dev/unreleased/_index.md b/site/content/in-dev/unreleased/_index.md index 023d3b0f72..183641f706 100644 --- a/site/content/in-dev/unreleased/_index.md +++ b/site/content/in-dev/unreleased/_index.md @@ -172,7 +172,7 @@ Polaris RBAC uses two different role types to delegate privileges: - **Catalog roles:** Configured with certain privileges on Polaris catalog resources and granted to principal roles. -For more information, see [Access control]({{% ref "access-control" %}}). +For more information, see [Access control]({{% ref "managing-security/access-control" %}}). ## Legal Notices diff --git a/site/content/in-dev/unreleased/getting-started/_index.md b/site/content/in-dev/unreleased/getting-started/_index.md index ee0f9b6871..1707ceacd2 100644 --- a/site/content/in-dev/unreleased/getting-started/_index.md +++ b/site/content/in-dev/unreleased/getting-started/_index.md @@ -17,9 +17,23 @@ # specific language governing permissions and limitations # under the License. # -title: 'Getting Started' +title: Getting Started with Apache Polaris +linkTitle: Getting Started type: docs weight: 101 -build: - render: never --- + +The fastest way to get started is with our Docker Compose examples. Each example provides a complete working environment with detailed instructions. + +## Next Steps + +1. Check/Install dependencies +2. Choose the way you want to deploy Polaris +3. Create a catalog +4. Check Using polaris page + +## Getting Help + +- Documentation: https://polaris.apache.org +- GitHub Issues: https://github.com/apache/polaris/issues +- Slack: [Join Apache Polaris Community](https://join.slack.com/t/apache-polaris/shared_invite/zt-2y3l3r0fr-VtoW42ltir~nSzCYOrQgfw) diff --git a/site/content/in-dev/unreleased/getting-started/creating-a-catalog/_index.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/_index.md new file mode 100644 index 0000000000..eeaf431733 --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/_index.md @@ -0,0 +1,54 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Creating a Catalog +linkTitle: Creating a Catalog +type: docs +weight: 300 +--- + +The following Object Storage providers can be configured as storage backends for your Polaris catalog: + +- [S3 compatible object stores]({{< ref "s3.md" >}}) +- [Google Cloud Storage]({{< ref "catalog-gcs.md" >}}) +- [Azure Blob Storage]({{< ref "catalog-azure.md" >}}) +- Local file system (By default for testing only) + + +## Create a catalog using polaris CLI + +Check full list of options for the `polaris catalogs create` command [here]({{% ref "../../command-line-interface#create" %}}) + +### Example + +```shell +CLIENT_ID=root \ +CLIENT_SECRET=s3cr3t \ +DEFAULT_BASE_LOCATION=s3://example-bucket/my_data \ +ROLE_ARN=arn:aws:iam::111122223333:role/ExampleCorpRole \ +./polaris \ + --client-id ${CLIENT_ID} \ + --client-secret ${CLIENT_SECRET} \ + catalogs \ + create \ + --storage-type s3 \ + --default-base-location ${DEFAULT_BASE_LOCATION} \ + --role-arn ${ROLE_ARN} \ + my_catalog +``` diff --git a/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-azure.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-azure.md new file mode 100644 index 0000000000..8666f28876 --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-azure.md @@ -0,0 +1,55 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Creating a catalog on Azure +linkTitle: Azure +type: docs +weight: 300 +--- + +For the `polaris catalogs create` [command]({{% ref "../../command-line-interface#create" %}}) there are few `azure` only options + +```text +--storage-type azure +--tenant-id (Required for Azure) A tenant ID to use when connecting to Azure Storage +--multi-tenant-app-name (Only for Azure) The app name to use when connecting to Azure Storage +--consent-url (Only for Azure) A consent URL granting permissions for the Azure Storage location +``` + +### example + +```shell +CLIENT_ID=root \ +CLIENT_SECRET=s3cr3t \ +DEFAULT_BASE_LOCATION=abfss://tenant123@blob.core.windows.net \ +TENANT_ID=tenant123.onmicrosoft.com \ +MULTI_TENANT_APP_NAME=myapp \ +CONSENT_URL=https://myapp.com/consent +./polaris \ + --client-id ${CLIENT_ID} \ + --client-secret ${CLIENT_SECRET} \ + catalogs \ + create \ + --storage-type azure \ + --tenant-id ${TENANT_ID} \ + --multi-tenant-app-name ${MULTI_TENANT_APP_NAME} \ + --consent-url ${CONSENT_URL} \ + --default-base-location ${DEFAULT_BASE_LOCATION} \ + my_azure_catalog +``` \ No newline at end of file diff --git a/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-gcs.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-gcs.md new file mode 100644 index 0000000000..db6214e38c --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/catalog-gcs.md @@ -0,0 +1,49 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Creating a catalog on Google Cloud Storage (GCS) +linkTitle: GCS +type: docs +weight: 200 +--- + +For the `polaris catalogs create` [command]({{% ref "../../command-line-interface#create" %}}) there are few `gcs` only options + +```text +--storage-type gcs +--service-account (Only for GCS) The service account to use when connecting to GCS +``` + +### example + +```shell +CLIENT_ID=root \ +CLIENT_SECRET=s3cr3t \ +DEFAULT_BASE_LOCATION=gs://my-ml-bucket/predictions/ \ +SERVICE_ACCOUNT=serviceAccount:my-service-account@my-project.iam.gserviceaccount.com \ +./polaris \ + --client-id ${CLIENT_ID} \ + --client-secret ${CLIENT_SECRET} \ + catalogs \ + create \ + --storage-type gcs \ + --service-account ${SERVICE_ACCOUNT} \ + --default-base-location ${DEFAULT_BASE_LOCATION} \ + my_gcs_catalog +``` \ No newline at end of file diff --git a/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/_index.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/_index.md new file mode 100644 index 0000000000..2bf78106f8 --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/_index.md @@ -0,0 +1,38 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Creating a catalog on S3 compatible cloud providers +linkTitle: S3 +type: docs +weight: 100 +--- + +The following S3 compatible cloud providers can be configured as storage backends for your Polaris catalog: + +- [AWS S3]({{< ref "catalog-aws.md" >}}) +- [MinIO]({{< ref "catalog-minio.md" >}}) + +For the `polaris catalogs create` [command]({{% ref "../../../command-line-interface#create" %}}) there are few `s3` only options + +```text +--storage-type s3 +--role-arn (Required for S3) A role ARN to use when connecting to S3 +--region (Only for S3) The region to use when connecting to S3 +--external-id (Only for S3) The external ID to use when connecting to S3 +``` diff --git a/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-aws.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-aws.md new file mode 100644 index 0000000000..5da41de0f5 --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-aws.md @@ -0,0 +1,47 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Creating a catalog on AWS S3 +linkTitle: AWS +type: docs +weight: 100 +--- + + +### example + +```shell +CLIENT_ID=root \ +CLIENT_SECRET=s3cr3t \ +DEFAULT_BASE_LOCATION=s3://example-bucket/my_data \ +ROLE_ARN=arn:aws:iam::111122223333:role/ExampleCorpRole \ +REGION=us-west-2 \ +EXTERNAL_ID=12345678901234567890 \ +./polaris \ + --client-id ${CLIENT_ID} \ + --client-secret ${CLIENT_SECRET} \ + catalogs \ + create \ + --storage-type s3 \ + --default-base-location ${DEFAULT_BASE_LOCATION} \ + --role-arn ${ROLE_ARN} \ + --region ${REGION} \ + --external-id ${EXTERNAL_ID} \ + my_aws_catalog +``` \ No newline at end of file diff --git a/site/content/in-dev/unreleased/getting-started/minio.md b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-minio.md similarity index 95% rename from site/content/in-dev/unreleased/getting-started/minio.md rename to site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-minio.md index 3eda7db622..18491617cd 100644 --- a/site/content/in-dev/unreleased/getting-started/minio.md +++ b/site/content/in-dev/unreleased/getting-started/creating-a-catalog/s3/catalog-minio.md @@ -17,9 +17,10 @@ # specific language governing permissions and limitations # under the License. # -Title: Deploying Polaris on MinIO +title: Creating a catalog on MinIO +linkTitle: MinIO type: docs -weight: 350 +weight: 200 --- In this guide we walk through setting up a simple Polaris Server with local [MinIO](https://www.min.io/) storage. @@ -42,7 +43,7 @@ docker compose -f getting-started/minio/docker-compose.yml up ``` The compose script will start MinIO on default ports (API on 9000, UI on 9001) -plus a Polaris Server pre-configured to that MinIO instance. +plus a Polaris Server pre-configured to that MinIO instance. In this example the `root` principal has its password set to `s3cr3t`. @@ -86,7 +87,7 @@ mc ls pol/bucket123/ns/t1 [2025-08-13 18:52:38 EDT] 0B metadata/ ``` -Note: the values of `minio_root`, `m1n1opwd` and `bucket123` are defined in the docker compose file. +Note: the values of `minio_root`, `m1n1opwd` and `bucket123` are defined in the docker compose file. # Notes on Storage Configuation diff --git a/site/content/in-dev/unreleased/getting-started/deploying-polaris/_index.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/_index.md index c6b293d29f..e975f69274 100644 --- a/site/content/in-dev/unreleased/getting-started/deploying-polaris/_index.md +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/_index.md @@ -17,11 +17,13 @@ # specific language governing permissions and limitations # under the License. # -Title: Deploying Polaris on Cloud Providers +title: Deploying Polaris +linkTitle: Deploying Polaris type: docs -weight: 300 +weight: 200 --- -We will now demonstrate how to deploy Polaris locally, as well as with all supported Cloud Providers: Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP). +Here you can find the guides of how to deploy Polaris locally, as well as with all supported Cloud Providers: Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP). -Locally, Polaris can be deployed using both Docker and local build. On the cloud, this tutorial will deploy Polaris using Docker only - but local builds can also be executed. +Locally, Polaris can be deployed using both docker and local build. +On the cloud, the following tutorials will deploy Polaris using docker environment. diff --git a/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/_index.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/_index.md new file mode 100644 index 0000000000..56626bc2c3 --- /dev/null +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/_index.md @@ -0,0 +1,27 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Deploying Polaris on Cloud Providers +linkTitle: Cloud Providers +type: docs +weight: 300 +--- + +Polaris can be deployed on various cloud providers, including Amazon Web Services (AWS), Azure, and Google Cloud Platform (GCP). +In the following guides, we will walk you through the process of deploying Polaris on each of these cloud providers. diff --git a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-aws.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-aws.md similarity index 94% rename from site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-aws.md rename to site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-aws.md index 832cc67bcf..13933f25ee 100644 --- a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-aws.md +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-aws.md @@ -17,7 +17,8 @@ # specific language governing permissions and limitations # under the License. # -Title: Deploying Polaris on Amazon Web Services (AWS) +title: Deploying Polaris on Amazon Web Services (AWS) +linkTitle: AWS type: docs weight: 310 --- @@ -44,7 +45,7 @@ export CLIENT_SECRET=s3cr3t ``` ## Next Steps -Congrats, you now have a running instance of1 Polaris! For details on how to use Polaris, check out the [Using Polaris]({{% relref "../using-polaris" %}}) page. +Congrats, you now have a running instance of1 Polaris! For details on how to use Polaris, check out the [Using Polaris]({{% relref "../../using-polaris" %}}) page. ## Cleanup Instructions To shut down the Polaris server, run the following commands: @@ -54,4 +55,4 @@ export ASSETS_PATH=$(pwd)/getting-started/assets/ docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml down ``` -To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../configuring-polaris-for-production" %}}) page. +To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../../configuring-polaris-for-production" %}}) page. diff --git a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-azure.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-azure.md similarity index 80% rename from site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-azure.md rename to site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-azure.md index da60198b0f..cdb911b3ab 100644 --- a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-azure.md +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-azure.md @@ -17,12 +17,13 @@ # specific language governing permissions and limitations # under the License. # -Title: Deploying Polaris on Azure +title: Deploying Polaris on Azure +linkTitle: Azure type: docs weight: 320 --- -Build and launch Polaris using the AWS Startup Script at the location provided in the command below. This script will start an [Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/overview) instance, which will be used as the backend Postgres instance holding all Polaris data. +Build and launch Polaris using the Azure Startup Script at the location provided in the command below. This script will start an [Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/overview) instance, which will be used as the backend Postgres instance holding all Polaris data. Additionally, Polaris will be bootstrapped to use this database and Docker containers will be spun up for Spark SQL and Trino. The requirements to run the script below are: @@ -39,7 +40,7 @@ export CLIENT_SECRET=s3cr3t ``` ## Next Steps -Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% relref "../using-polaris" %}}) page. +Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% relref "../../using-polaris" %}}) page. ## Cleanup Instructions To shut down the Polaris server, run the following commands: @@ -49,4 +50,4 @@ export ASSETS_PATH=$(pwd)/getting-started/assets/ docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml down ``` -To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../configuring-polaris-for-production" %}}) page. +To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../../configuring-polaris-for-production" %}}) page. diff --git a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-gcp.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-gcp.md similarity index 90% rename from site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-gcp.md rename to site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-gcp.md index 30c99c61ba..d50a2c9d3a 100644 --- a/site/content/in-dev/unreleased/getting-started/deploying-polaris/quickstart-deploy-gcp.md +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/cloud-deploy/deploy-gcp.md @@ -17,12 +17,13 @@ # specific language governing permissions and limitations # under the License. # -Title: Deploying Polaris on Google Cloud Platform (GCP) +title: Deploying Polaris on Google Cloud Platform (GCP) +linkTitle: GCP type: docs weight: 330 --- -Build and launch Polaris using the AWS Startup Script at the location provided in the command below. This script will start a [Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres) instance, which will be used as the backend Postgres instance holding all Polaris data. +Build and launch Polaris using the GCP Startup Script at the location provided in the command below. This script will start a [Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres) instance, which will be used as the backend Postgres instance holding all Polaris data. Additionally, Polaris will be bootstrapped to use this database and Docker containers will be spun up for Spark SQL and Trino. The requirements to run the script below are: @@ -39,7 +40,7 @@ export CLIENT_SECRET=s3cr3t ``` ## Next Steps -Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% relref "../using-polaris" %}}) page. +Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% relref "../../using-polaris" %}}) page. ## Cleanup Instructions To shut down the Polaris server, run the following commands: @@ -49,4 +50,4 @@ export ASSETS_PATH=$(pwd)/getting-started/assets/ docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml down ``` -To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../configuring-polaris-for-production" %}}) page. +To deploy Polaris in a production setting, please review further recommendations at the [Configuring Polaris for Production]({{% relref "../../../configuring-polaris-for-production" %}}) page. diff --git a/site/content/in-dev/unreleased/getting-started/quickstart.md b/site/content/in-dev/unreleased/getting-started/deploying-polaris/local-deploy.md similarity index 92% rename from site/content/in-dev/unreleased/getting-started/quickstart.md rename to site/content/in-dev/unreleased/getting-started/deploying-polaris/local-deploy.md index 6d92c1635a..8c9d6afc12 100644 --- a/site/content/in-dev/unreleased/getting-started/quickstart.md +++ b/site/content/in-dev/unreleased/getting-started/deploying-polaris/local-deploy.md @@ -17,7 +17,8 @@ # specific language governing permissions and limitations # under the License. # -Title: Quickstart +title: Deploying Polaris locally +linkTitle: Local deployment type: docs weight: 200 --- @@ -89,7 +90,7 @@ INFO [io.quarkus] [,] [,,,] (main) Installed features: [...] At this point, Polaris is running. When using a Gradle-launched Polaris instance in this tutorial, we'll launch an instance of Polaris that stores entities only in-memory. This means that any entities that you define will be destroyed when Polaris is shut down. -For more information on how to configure Polaris for production usage, see the [docs]({{% relref "../configuring-polaris-for-production" %}}). +For more information on how to configure Polaris for production usage, see the [docs]({{% relref "../../configuring-polaris-for-production" %}}). When Polaris is run using the `./gradlew run` command, the root principal credentials are `root` and `s3cr3t` for the `CLIENT_ID` and `CLIENT_SECRET`, respectively. @@ -97,7 +98,7 @@ When Polaris is run using the `./gradlew run` command, the root principal creden #### Apache Spark -If you want to connect to Polaris with [Apache Spark](https://spark.apache.org/), you'll need to start by cloning Spark. As in the [prerequisites]({{% ref "install-dependencies#git" %}}), make sure [git](https://git-scm.com/) is installed first. +If you want to connect to Polaris with [Apache Spark](https://spark.apache.org/), you'll need to start by cloning Spark. As in the [prerequisites]({{% ref "../install-dependencies#git" %}}), make sure [git](https://git-scm.com/) is installed first. Then, clone Spark and check out a versioned branch. This guide uses [Spark 3.5](https://spark.apache.org/releases/spark-release-3-5-0.html). @@ -106,11 +107,11 @@ git clone --branch branch-3.5 https://github.com/apache/spark.git ~/spark ``` #### Trino -If you want to connect to Polaris with [Trino](https://trino.io/), it is recommended to set up a test instance of Trino using Docker. As in the [prerequisites]({{% ref "install-dependencies#docker" %}}), make sure [Docker](https://www.docker.com/) is installed first +If you want to connect to Polaris with [Trino](https://trino.io/), it is recommended to set up a test instance of Trino using Docker. As in the [prerequisites]({{% ref "../install-dependencies#docker" %}}), make sure [Docker](https://www.docker.com/) is installed first ```shell docker run --name trino -d -p 8080:8080 trinodb/trino ``` ## Next Steps -Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% ref "using-polaris" %}}) page. +Congrats, you now have a running instance of Polaris! For further information regarding how to use Polaris, check out the [Using Polaris]({{% ref "../using-polaris" %}}) page. diff --git a/site/content/in-dev/unreleased/managing-security/_index.md b/site/content/in-dev/unreleased/managing-security/_index.md new file mode 100644 index 0000000000..d96c870c54 --- /dev/null +++ b/site/content/in-dev/unreleased/managing-security/_index.md @@ -0,0 +1,24 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Managing Security +linkTitle: Managing Security +type: docs +weight: 1001 +--- diff --git a/site/content/in-dev/unreleased/access-control.md b/site/content/in-dev/unreleased/managing-security/access-control.md similarity index 99% rename from site/content/in-dev/unreleased/access-control.md rename to site/content/in-dev/unreleased/managing-security/access-control.md index 727b4e60f3..b8a1b697ca 100644 --- a/site/content/in-dev/unreleased/access-control.md +++ b/site/content/in-dev/unreleased/managing-security/access-control.md @@ -17,9 +17,10 @@ # specific language governing permissions and limitations # under the License. # -Title: Access Control +title: Role-Based Access Control +linkTitle: Access Control type: docs -weight: 500 +weight: 200 --- This section provides information about how access control works for Apache Polaris (Incubating). diff --git a/site/content/in-dev/unreleased/external-idp.md b/site/content/in-dev/unreleased/managing-security/external-idp/_index.md similarity index 98% rename from site/content/in-dev/unreleased/external-idp.md rename to site/content/in-dev/unreleased/managing-security/external-idp/_index.md index 01b465c641..ff286b9f00 100644 --- a/site/content/in-dev/unreleased/external-idp.md +++ b/site/content/in-dev/unreleased/managing-security/external-idp/_index.md @@ -17,9 +17,10 @@ # specific language governing permissions and limitations # under the License. # -title: External Identity Providers +title: Identity Providers +linkTitle: Identity Providers type: docs -weight: 550 +weight: 300 --- Apache Polaris supports authentication via external identity providers (IdPs) using OpenID Connect (OIDC) in addition to the internal authentication system. This feature enables flexible identity federation with enterprise IdPs and allows gradual migration or hybrid authentication strategies across realms in Polaris. @@ -116,7 +117,7 @@ External authentication is configured via Quarkus OIDC and Polaris-specific OIDC ### OIDC Tenant Configuration -At least one OIDC tenant must be explicitly enabled. In Polaris, realms and OIDC tenants are distinct concepts. An OIDC tenant represents a specific identity provider configuration (e.g., `quarkus.oidc.idp1`). A [realm]({{% ref "realm" %}}) is a logical partition within Polaris. +At least one OIDC tenant must be explicitly enabled. In Polaris, realms and OIDC tenants are distinct concepts. An OIDC tenant represents a specific identity provider configuration (e.g., `quarkus.oidc.idp1`). A [realm]({{% ref "../../realm" %}}) is a logical partition within Polaris. - Multiple realms can share a single OIDC tenant. - Each realm can be associated with only one OIDC tenant. @@ -344,4 +345,7 @@ When internal authentication is enabled, Polaris uses token brokers to handle th - Result Polaris roles: `PRINCIPAL_ROLE:service_admin` and `PRINCIPAL_ROLE:catalog_admin` - + +### Additional Examples + +For complete Keycloak integration example, see: [Keycloak External IDP Configuration Guide]({{< relref "keycloak-idp.md" >}}) \ No newline at end of file diff --git a/site/content/in-dev/unreleased/managing-security/external-idp/keycloak-idp.md b/site/content/in-dev/unreleased/managing-security/external-idp/keycloak-idp.md new file mode 100644 index 0000000000..1d4657c38f --- /dev/null +++ b/site/content/in-dev/unreleased/managing-security/external-idp/keycloak-idp.md @@ -0,0 +1,210 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Using Polaris with Keycloak as external IDP +linkTitle: Using Keycloak IDP +type: docs +weight: 100 +--- + +This example uses Keycloak as an **external** identity provider for Polaris. The "iceberg" realm is automatically +created and configured from the `iceberg-realm.json` file. + +This Keycloak realm contains 1 client definition: `client1:s3cr3t`. It is configured to return tokens with the following +fixed claims: + +- `principal_id`: the principal ID of the user. It is always set to zero (0) in this example. +- `principal_name`: the principal name of the user. It is always set to "root" in this example. +- `principal_roles`: the principal roles of the user. It is always set to `["server_admin", "catalog_admin"]` in this + example. + +This is obviously not a realistic configuration. In a real-world scenario, you would configure Keycloak to return the +actual principal ID, name and roles of the user. Note that principals and principal roles must have been created in +Polaris beforehand, and the principal ID, name and roles must match the ones returned by Keycloak. + +Polaris is configured with 3 realms: + +- `realm-internal`: This is the default realm, and is configured to use the internal authentication only. It accepts + token issues by Polaris itself only. +- `realm-external`: This realm is configured to use an external identity provider (IDP) for authentication only. It + accepts tokens issued by Keycloak only. +- `realm-mixed`: This realm is configured to use both the internal and external authentication. It accepts tokens + issued by both Polaris and Keycloak. + +For more information about how to configure Polaris with external authentication, see the +[Polaris documentation](https://polaris.apache.org/in-dev/unreleased/external-idp/). + +## Starting the Example + +1. Build the Polaris server image if it's not already present locally: + + ```shell + ./gradlew \ + :polaris-server:assemble \ + :polaris-server:quarkusAppPartsBuild --rerun \ + -Dquarkus.container-image.build=true + ``` + +2. Start the docker compose group by running the following command from the root of the repository: + + ```shell + docker compose -f getting-started/keycloak/docker-compose.yml up + ``` + +## Requesting a Token + +Note: the commands below require `jq` to be installed on your machine. + +### From Polaris + +You can request a token from Polaris for realms `realm-internal` and `realm-mixed`: + +1. Open a terminal and run the following command to request an access token for the `realm-internal` realm: + + ```shell + polaris_token_realm_internal=$(curl -s http://localhost:8181/api/catalog/v1/oauth/tokens \ + --user root:s3cr3t \ + -H 'Polaris-Realm: realm-internal' \ + -d 'grant_type=client_credentials' \ + -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r .access_token) + ``` + + This token is valid only for the `realm-internal` realm. + +2. Open a terminal and run the following command to request an access token for the `realm-mixed` realm: + + ```shell + polaris_token_realm_mixed=$(curl -s http://localhost:8181/api/catalog/v1/oauth/tokens \ + --user root:s3cr3t \ + -H 'Polaris-Realm: realm-mixed' \ + -d 'grant_type=client_credentials' \ + -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r .access_token) + ``` + + This token is valid only for the `realm-mixed` realm. + +Polaris tokens are valid for 1 hour. + +Note: if you request a Polaris token for the `realm-external` realm, it will not work because Polaris won't issue tokens +for this realm: + +```shell +curl -v http://localhost:8181/api/catalog/v1/oauth/tokens \ + --user root:s3cr3t \ + -H 'Polaris-Realm: realm-external' \ + -d 'grant_type=client_credentials' \ + -d 'scope=PRINCIPAL_ROLE:ALL' +``` + +This will return a `501 Not Implemented` error because for this realm, the internal token endpoint has been deactivated. + +### From Keycloak + +You can request a token from Keycloak for the `realm-external` and `realm-mixed` realms: + +1. Open a terminal and run the following command to request an access token from Keycloak: + + ```shell + keycloak_token=$(curl -s http://keycloak:8080/realms/iceberg/protocol/openid-connect/token \ + --resolve keycloak:8080:127.0.0.1 \ + --user client1:s3cr3t \ + -d 'grant_type=client_credentials' | jq -r .access_token) + ``` + +Note the `--resolve` option: it is used to send the request with the `Host` header set to `keycloak`. This is necessary +because Keycloak issues tokens with the `iss` claim matching the request's `Host` header; without this, the token would +not be valid when used against Polaris because the `iss` claim would be `127.0.0.1`, but Polaris expects it to be +`keycloak`, since that's Keycloak's hostname within the Docker network. + +Tokens issued by Keycloak can be used to access Polaris with the `realm-external` or `realm-mixed` realms. Access tokens +are valid for 1 hour. + +You can also access the Keycloak admin console. Open a browser and go to [http://localhost:8080](http://localhost:8080), +then log in with the username `admin` and password `admin` (you can change this in the docker-compose file). + +## Accessing Polaris with the Tokens + +You can access Polaris using the tokens you obtained above. The following examples show how to use the tokens with +`curl`: + +### Using the Polaris Token + +1. Open a terminal and run the following command to list the principal roles in the `realm-internal` realm: + + ```shell + curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $polaris_token_realm_internal" \ + -H 'Polaris-Realm: realm-internal' \ + -H 'Accept: application/json' + ``` + +2. Open a terminal and run the following command to list the principal roles in the `realm-mixed` realm: + + ```shell + curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $polaris_token_realm_mixed" \ + -H 'Polaris-Realm: realm-mixed' \ + -H 'Accept: application/json' + ``` + +Note: you cannot mix tokens from different realms. For example, you cannot use a token from the `realm-internal` realm to access +the `realm-mixed` realm: + +```shell +curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $polaris_token_realm_internal" \ + -H 'Polaris-Realm: realm-mixed' \ + -H 'Accept: application/json' +``` + +This will return a `401 Unauthorized` error because the token is not valid for the `realm-mixed` realm. + +### Using the Keycloak Token + +The same Keycloak token can be used to access both the `realm-external` and `realm-mixed` realms, as it is valid for +both (both realms share the same OIDC tenant configuration). + +1. Open a terminal and run the following command to list the principal roles in the `realm-external` realm: + + ```shell + curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $keycloak_token" \ + -H 'Polaris-Realm: realm-external' \ + -H 'Accept: application/json' + ``` + +2. Open a terminal and run the following command to list the principal roles in the `realm-mixed` realm: + + ```shell + curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $keycloak_token" \ + -H 'Polaris-Realm: realm-mixed' \ + -H 'Accept: application/json' + ``` + +Note: you cannot use a Keycloak token to access the `realm-internal` realm: + +```shell +curl -v http://localhost:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer $keycloak_token" \ + -H 'Polaris-Realm: realm-internal' \ + -H 'Accept: application/json' +``` + +This will return a `401 Unauthorized` error because the token is not valid for the `realm-internal` realm. \ No newline at end of file