From 9a916739fbcf7b71f6569124bac36eca500a5de1 Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 11:16:01 -0700 Subject: [PATCH 01/23] wip --- docs/configuring-polaris-for-production.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 docs/configuring-polaris-for-production.md diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md new file mode 100644 index 0000000000..e69de29bb2 From 7c01fec0b1c8377f251b4d437b0b3e1f7486a9a1 Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 13:01:01 -0700 Subject: [PATCH 02/23] checkin --- docs/configuring-polaris-for-production.md | 31 ++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md index e69de29bb2..28c3047954 100644 --- a/docs/configuring-polaris-for-production.md +++ b/docs/configuring-polaris-for-production.md @@ -0,0 +1,31 @@ +# Configuring Polaris for Production + +The default `polaris-server.yml` configuration is intended for develoment and testing. When deploying Polaris in production, there are several best practices to keep in mind. + +## Security + +### Configurations + +There are many Polaris configurations that should be adjusted to ensure a secure Polaris deployment. Some of these configurations are briefly outlined below, along with a short description of each. + +* **oauth2** + - Configure [OAuth](https://oauth.net/2/) with this setting, including a token broker + +* **callContextResolver** & **realmContextResolver** + - Use these configurations to specify a service that can resolve a realm from your bearer tokens. + - The service(s) used here must implement the relevant interfaces (e.g. [CallContextResolver](https://github.com/polaris-catalog/polaris/blob/8290019c10290a600e40b35ddb1e2f54bf99e120/polaris-service/src/main/java/io/polaris/service/context/CallContextResolver.java#L27)). + +* **authenticator.tokenBroker** + - Ensure that this setting reflects the token broker specified in **oauth2** above + + + +## Other Configurations + +When deploying Polaris in production, consider adjusting the following configurations: + +* **featureConfiguration.SUPPORTED_CATALOG_STORAGE_TYPES** + - By default, the `FILE` storage type may be supported. This is intended for testing, and in produciton you'll likely want to disable it + - Here you can also disable or enable any other storage type based on your expected usage of Apache Iceberg + + From c8be49f010de2eb5fe1a29e801e3d4f8f0170f09 Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 13:09:45 -0700 Subject: [PATCH 03/23] check in --- docs/configuring-polaris-for-production.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md index 28c3047954..6f0095871a 100644 --- a/docs/configuring-polaris-for-production.md +++ b/docs/configuring-polaris-for-production.md @@ -1,3 +1,20 @@ + + + # Configuring Polaris for Production The default `polaris-server.yml` configuration is intended for develoment and testing. When deploying Polaris in production, there are several best practices to keep in mind. From 02f320f7699c5b41b0e77e5fcb5c0a68394131fc Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 13:19:05 -0700 Subject: [PATCH 04/23] rebase --- docs/configuring-polaris-for-production.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md index 6f0095871a..50c06191a8 100644 --- a/docs/configuring-polaris-for-production.md +++ b/docs/configuring-polaris-for-production.md @@ -35,8 +35,22 @@ There are many Polaris configurations that should be adjusted to ensure a secure * **authenticator.tokenBroker** - Ensure that this setting reflects the token broker specified in **oauth2** above +* **cors** + - Use settings like **cors.allowed-origins** to control what connections your Polaris service will accept +## Metastore Management + +Use the configuration `metaStoreManager` to configure a [MetastoreManager](https://github.com/polaris-catalog/polaris/blob/627dc602eb15a3258dcc32babf8def34cf6de0e9/polaris-core/src/main/java/io/polaris/core/persistence/PolarisMetaStoreManager.java#L47) implementation where Polaris entities will be persisted. + +The default `in-memory` implementation is meant for testing and not suitable for production usage. Instead, consider an implementation such as `eclipse-link` which allows you to store metadata in a remote database. + +Credentials and metadata will be stored in the metastore manager, and so be sure that your metastore manager is accordingly secured. + +### Configuring EclipseLink + +To use [EclipseLink](https://eclipse.dev/eclipselink/) for metastore management, specify the configuration `metaStoreManager.conf-file` to point to an [EclipseLink `persistence.xml` file](https://eclipse.dev/eclipselink/documentation/2.5/solutions/testingjpa002.htm). This file, local to the Polaris service, will contain information on what database to use for metastore management and how to connect to it. + ## Other Configurations When deploying Polaris in production, consider adjusting the following configurations: From 0b4e492b9ed05cf91c4c49aa4f86cf6097d49cdb Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 13:36:46 -0700 Subject: [PATCH 05/23] bootstrapping notes --- docs/configuring-polaris-for-production.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md index 50c06191a8..a6c19b9d19 100644 --- a/docs/configuring-polaris-for-production.md +++ b/docs/configuring-polaris-for-production.md @@ -51,6 +51,22 @@ Credentials and metadata will be stored in the metastore manager, and so be sure To use [EclipseLink](https://eclipse.dev/eclipselink/) for metastore management, specify the configuration `metaStoreManager.conf-file` to point to an [EclipseLink `persistence.xml` file](https://eclipse.dev/eclipselink/documentation/2.5/solutions/testingjpa002.htm). This file, local to the Polaris service, will contain information on what database to use for metastore management and how to connect to it. +### Bootstrapping + +Before using Polaris when using a metastore manager other than `in-memory`, you must **bootstrap** the metastore manager. This is a manual operation that must be performed once in order to prepare the metastore manager to integrate with Polaris. When the metastore manager is bootstrapped, any existing Polaris entities in the metastore manager may be purged. + +To bootstrap Polaris, run a command like the following: + +```bash +java jar /app/snowflake-polaris-all.jar bootstrap polaris-server.yml +``` + +Afterwards, you can launch Polaris normally with a command like the following: + +```bash +java jar /app/snowflake-polaris-all.jar server polaris-server.yml +``` + ## Other Configurations When deploying Polaris in production, consider adjusting the following configurations: From ba2466ad872881e9bee7676307913f8db3c7659b Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 14:05:52 -0700 Subject: [PATCH 06/23] emphasis --- docs/configuring-polaris-for-production.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md index a6c19b9d19..f35be324c1 100644 --- a/docs/configuring-polaris-for-production.md +++ b/docs/configuring-polaris-for-production.md @@ -53,7 +53,7 @@ To use [EclipseLink](https://eclipse.dev/eclipselink/) for metastore management, ### Bootstrapping -Before using Polaris when using a metastore manager other than `in-memory`, you must **bootstrap** the metastore manager. This is a manual operation that must be performed once in order to prepare the metastore manager to integrate with Polaris. When the metastore manager is bootstrapped, any existing Polaris entities in the metastore manager may be purged. +Before using Polaris when using a metastore manager other than `in-memory`, you must **bootstrap** the metastore manager. This is a manual operation that must be performed **only once** in order to prepare the metastore manager to integrate with Polaris. When the metastore manager is bootstrapped, any existing Polaris entities in the metastore manager may be **purged**. To bootstrap Polaris, run a command like the following: From 94b6dee20a9d71e017a98eae5f4584a19b761c29 Mon Sep 17 00:00:00 2001 From: Eric Maynard Date: Wed, 31 Jul 2024 14:37:05 -0700 Subject: [PATCH 07/23] plug the quickstart in the readme, so there's a path to the other docs --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 44d6c7a469..a3c66067d6 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,9 @@ docker run -p 8080:80 -v ${PWD}:/spec redocly/cli build-docs spec/index.yaml --o Command-Line getting started ------------------- + +_For an end-to-end tutorial on building, deploying, and managing entities in a local Polaris instance see the [quickstart guide](./docs/quickstart.md)._ + Polaris is a multi-module project with three modules: - `polaris-core` - The main Polaris entity definitions and core business logic From 1b0c0499a4c68e08138b43e151eb41a89b6f53e0 Mon Sep 17 00:00:00 2001 From: annafil <7892219+annafil@users.noreply.github.com> Date: Thu, 1 Aug 2024 09:23:18 -0700 Subject: [PATCH 08/23] ensure new links are rendered in web --- docs/index.html | 142 ++++++++++++++++++++++++++++++++++++++++++++---- spec/docs.yaml | 3 + spec/index.yaml | 137 +++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 268 insertions(+), 14 deletions(-) diff --git a/docs/index.html b/docs/index.html index ac8987d906..b7a80094a4 100644 --- a/docs/index.html +++ b/docs/index.html @@ -430,7 +430,7 @@ -
Polaris Catalog Logo
API docs by Redocly
Polaris Catalog Logo
API docs by Redocly

Polaris Catalog Documentation

Download OpenAPI specification:Download

E-mail: community [at] polaris.io URL: https://github.com/polaris-catalog/polaris License: Apache v2.0

Quick Start

This guide serves as a introduction to several key entities that can be managed with Polaris, describes how to build and deploy Polaris locally, and finally includes examples of how to use Polaris with Spark and Trino.

@@ -1295,7 +1291,6 @@

Access control

For more information, see Access control.

Polaris Catalog Entities

Access control WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - --> <p>This page documents various entities that can be managed in Polaris.</p> ">

This page documents various entities that can be managed in Polaris.

@@ -1968,6 +1960,134 @@

Catalog privileges

Diagram that shows an example of how RBAC works in Polaris Catalog.

+

Configuring Polaris for Production

+ + +

Configuring Polaris for Production

The default polaris-server.yml configuration is intended for develoment and testing. When deploying Polaris in production, there are several best practices to keep in mind.

+

Security

Configurations

+

There are many Polaris configurations that should be adjusted to ensure a secure Polaris deployment. Some of these configurations are briefly outlined below, along with a short description of each.

+
    +

  • oauth2

    +
      +
    • Configure OAuth with this setting, including a token broker
      • +
    +
    • +

  • callContextResolver & realmContextResolver

    +
      +
    • Use these configurations to specify a service that can resolve a realm from your bearer tokens.
      • +
    • The service(s) used here must implement the relevant interfaces (e.g. CallContextResolver).
      • +
    +
    • +

  • authenticator.tokenBroker

    +
      +
    • Ensure that this setting reflects the token broker specified in oauth2 above
      • +
    +
    • +

  • cors

    +
      +
    • Use settings like cors.allowed-origins to control what connections your Polaris service will accept
      • +
    +
    • +
+

Metastore Management

Use the configuration metaStoreManager to configure a MetastoreManager implementation where Polaris entities will be persisted.

+

The default in-memory implementation is meant for testing and not suitable for production usage. Instead, consider an implementation such as eclipse-link which allows you to store metadata in a remote database.

+

Credentials and metadata will be stored in the metastore manager, and so be sure that your metastore manager is accordingly secured.

+ +

To use EclipseLink for metastore management, specify the configuration metaStoreManager.conf-file to point to an EclipseLink persistence.xml file. This file, local to the Polaris service, will contain information on what database to use for metastore management and how to connect to it.

+

Bootstrapping

+

Before using Polaris when using a metastore manager other than in-memory, you must bootstrap the metastore manager. This is a manual operation that must be performed only once in order to prepare the metastore manager to integrate with Polaris. When the metastore manager is bootstrapped, any existing Polaris entities in the metastore manager may be purged.

+

To bootstrap Polaris, run a command like the following:

+
java jar /app/snowflake-polaris-all.jar bootstrap polaris-server.yml
+
+

Afterwards, you can launch Polaris normally with a command like the following:

+
java jar /app/snowflake-polaris-all.jar server polaris-server.yml
+
+

Other Configurations

When deploying Polaris in production, consider adjusting the following configurations:

+
    +
  • featureConfiguration.SUPPORTED_CATALOG_STORAGE_TYPES
      +
    • By default, the FILE storage type may be supported. This is intended for testing, and in produciton you'll likely want to disable it
      • +
    • Here you can also disable or enable any other storage type based on your expected usage of Apache Iceberg
      • +
    +
    • +

other

listCatalogs

List all catalogs in this polaris service

Authorizations:
Polaris_Management_Service_OAuth2

Responses

Authorizations:
Apache_Iceberg_REST_Catalog_API_BearerAuth
Request Body schema: application/x-www-form-urlencoded
required
Any of
grant_type
required
string
Value: "client_credentials"
scope
string
client_id
required
string
Authorizations:
Apache_Iceberg_REST_Catalog_API_BearerAuth
header Parameters
Authorization
string
Request Body schema: application/x-www-form-urlencoded
required
Any of
grant_type
required
string
Value: "client_credentials"
scope
string
client_id
required
string

Client ID

This can be sent in the request body, but OAuth2 recommends sending it in a Basic Authorization header.

@@ -3470,7 +3590,7 @@

Catalog privileges

" class="sc-euGpHm sc-exayXG fwfkcU jYGAQp">

Generic base server URL, with all parts configurable

{scheme}://{host}:{port}/{basePath}/v1/{prefix}/views/rename

Request samples

Content type
application/json
{
  • "source": {
    },
  • "destination": {
    }
}

Response samples

Content type
application/json
{
  • "error": {
    }
}