From c654dca31c02f3de702289230eb956956d7f35a6 Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Thu, 15 May 2025 14:54:41 -0700 Subject: [PATCH 1/6] Remove verbose terminal output for admin tool --- quarkus/admin/src/main/resources/application.properties | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/quarkus/admin/src/main/resources/application.properties b/quarkus/admin/src/main/resources/application.properties index 0d10451cf8..1828df6e85 100644 --- a/quarkus/admin/src/main/resources/application.properties +++ b/quarkus/admin/src/main/resources/application.properties @@ -23,7 +23,7 @@ # Cf. https://quarkus.io/guides/all-config quarkus.application.name=Apache Polaris Admin Tool (incubating) -quarkus.banner.path=/org/apache/polaris/admintool/banner.txt +quarkus.banner.enabled=false quarkus.container-image.build=false quarkus.container-image.push=false @@ -50,3 +50,7 @@ quarkus.index-dependency.guava.artifact-id=guava quarkus.index-dependency.protobuf.group-id=com.google.protobuf quarkus.index-dependency.protobuf.artifact-id=protobuf-java quarkus.datasource.devservices.image-name=postgres:17-alpine + +# ---- Logging Configuration ---- +# Set root logging level to WARNING to suppress INFO logs +quarkus.log.level=WARNING \ No newline at end of file From 640c0b14cee19bb4bdbb818432375a809ee93f01 Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Thu, 15 May 2025 17:11:40 -0700 Subject: [PATCH 2/6] Add doc for policy management --- site/content/in-dev/unreleased/policy.md | 200 +++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 site/content/in-dev/unreleased/policy.md diff --git a/site/content/in-dev/unreleased/policy.md b/site/content/in-dev/unreleased/policy.md new file mode 100644 index 0000000000..af96673af6 --- /dev/null +++ b/site/content/in-dev/unreleased/policy.md @@ -0,0 +1,200 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Policy +type: docs +weight: 425 +--- + +The Polaris Policy framework empowers organizations to centrally define, manage, and enforce fine-grained governance, lifecycle, and operational rules across all data resources in the catalog. + +With the policy API, you can: +- Create and manage policies +- Attach policies to specific resources (catalogs, namespaces, tables, or views) +- Check applicable policies for any given resource + +## What is a Policy? + +A policy in Apache Polaris is a structured object that defines rules governing actions on specified resources under +predefined conditions. Each policy contains: + +- **Name**: A unique identifier within a namespace +- **Type**: Determines the semantics and expected format of the policy content +- **Description**: Explains the purpose of the policy +- **Content**: Contains the actual rules defining the policy behavior +- **Version**: An automatically tracked revision number +- **Inheritable**: Whether the policy can be inherited by child resources, decided by its type + +### Policy Types + +Polaris supports several predefined system policy types (prefixed with `system.`): + +- **`system.data-compaction`**: Defines rules for data compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json + - Controls file compaction to optimize storage and query performance + +- **`system.metadata-compaction`**: Defines rules for metadata compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json + - Optimizes table metadata for improved performance + +- **`system.orphan-file-removal`**: Defines rules for removing orphaned files + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json + - Identifies and safely removes files that are no longer referenced by the table metadata + +- **`system.snapshot-expiry`**: Defines rules for snapshot expiration + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json + - Controls how long snapshots are retained before removal + +- **Custom policy types**: Can be defined for specific organizational needs (WIP) + +- **FGAC (Fine-Grained Access Control) policies**: Row filtering and column masking (WIP) + +### Policy Inheritance + +The object hierarchy in Polaris is structured as follows: + +``` + Catalog + | + Namespace + | + +-----+-----+ + | | + Table View +``` + +Policies can be attached at any level, and inheritance flows from catalog down to namespace, then to tables and views. + +Policies can be inheritable or non-inheritable: + +- **Inheritable policies**: Apply to the target resource and all its child resources +- **Non-inheritable policies**: Apply only to the specific target resource + +The inheritance follows an override mechanism: +1. Table-level policies override namespace and catalog policies +2. Namespace-level policies override parent namespace and catalog policies + +## Working with Policies + +### Creating a Policy + +To create a policy, you need to provide a name, type, and optionally a description and content: + +```json +POST /polaris/v1/{prefix}/namespaces/{namespace}/policies +{ + "name": "compaction-policy", + "type": "system.data-compaction", + "description": "Policy for optimizing table storage", + "content": "{\"version\": \"2025-02-03\", \"enable\": true, \"config\": {\"target_file_size_bytes\": 134217728}}" +} +``` + +The policy content is validated against a schema specific to its type. Here are a few policy content examples +- Data Compaction Policy +```json +{ + "version": "2025-02-03", + "enable": true, + "config": { + "target_file_size_bytes": 134217728, + "compaction_strategy": "bin-pack", + "max-concurrent-file-group-rewrites": 5 + } +} +``` +- Orphan File Removal Policy +```json +{ + "version": "2025-02-03", + "enable": true, + "max_orphan_file_age_in_days": 30, + "locations": ["s3://my-bucket/my-table-location"], + "config": { + "prefix_mismatch_mode": "ignore" + } +} +``` + +### Attaching Policies to Resources + +Policies can be attached to different resource levels: + +1. **Catalog level**: Applies to the entire catalog +2. **Namespace level**: Applies to a specific namespace +3. **Table-like level**: Applies to individual tables or views + +Example of attaching a policy to a table: + +```json +PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings +{ + "target": { + "type": "table-like", + "path": ["NS1", "NS2", "test_table_1"] + } +} +``` + +For inheritable policies, only one policy of a given type can be attached to a resource. For non-inheritable policies, multiple policies of the same type can be attached. + +### Retrieving Applicable Policies + +Here is an example to find all policies that apply to a specific resource (including inherited policies): +``` +GET /polaris/v1/catalog/applicable-policies?namespace=finance%1Fquarterly&target-name=transactions +``` + +**Sample response:** +```json +{ + "policies": [ + { + "name": "snapshot-expiry-policy", + "type": "system.snapshot-expiry", + "appliedAt": "namespace", + "content": { + "version": "2025-02-03", + "enable": true, + "config": { + "min_snapshot_to_keep": 1, + "max_snapshot_age_days": 2, + "max_ref_age_days": 3 + } + } + }, + { + "name": "compaction-policy", + "type": "system.data-compaction", + "appliedAt": "catalog", + "content": { + "version": "2025-02-03", + "enable": true, + "config": { + "target_file_size_bytes": 134217728 + } + } + } + ] +} +``` + +### API Reference + +For the complete and up-to-date API specification, see the [policy-api.yaml](https://github.com/apache/polaris/blob/main/spec/polaris-catalog-apis/policy-apis.yaml). \ No newline at end of file From 83592394f2c7352210cf99235f03962410124935 Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Thu, 15 May 2025 17:15:42 -0700 Subject: [PATCH 3/6] revert a change --- quarkus/admin/src/main/resources/application.properties | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/quarkus/admin/src/main/resources/application.properties b/quarkus/admin/src/main/resources/application.properties index 1828df6e85..0d10451cf8 100644 --- a/quarkus/admin/src/main/resources/application.properties +++ b/quarkus/admin/src/main/resources/application.properties @@ -23,7 +23,7 @@ # Cf. https://quarkus.io/guides/all-config quarkus.application.name=Apache Polaris Admin Tool (incubating) -quarkus.banner.enabled=false +quarkus.banner.path=/org/apache/polaris/admintool/banner.txt quarkus.container-image.build=false quarkus.container-image.push=false @@ -50,7 +50,3 @@ quarkus.index-dependency.guava.artifact-id=guava quarkus.index-dependency.protobuf.group-id=com.google.protobuf quarkus.index-dependency.protobuf.artifact-id=protobuf-java quarkus.datasource.devservices.image-name=postgres:17-alpine - -# ---- Logging Configuration ---- -# Set root logging level to WARNING to suppress INFO logs -quarkus.log.level=WARNING \ No newline at end of file From 60f80d79eba97ac560aa6b8d762c5278475e7261 Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Fri, 16 May 2025 11:20:55 -0700 Subject: [PATCH 4/6] Resolve comments --- site/content/in-dev/unreleased/policy.md | 27 +++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/site/content/in-dev/unreleased/policy.md b/site/content/in-dev/unreleased/policy.md index af96673af6..bdc0734556 100644 --- a/site/content/in-dev/unreleased/policy.md +++ b/site/content/in-dev/unreleased/policy.md @@ -31,7 +31,7 @@ With the policy API, you can: ## What is a Policy? -A policy in Apache Polaris is a structured object that defines rules governing actions on specified resources under +A policy in Apache Polaris is a structured entity that defines rules governing actions on specified resources under predefined conditions. Each policy contains: - **Name**: A unique identifier within a namespace @@ -52,38 +52,42 @@ Polaris supports several predefined system policy types (prefixed with `system.` - **`system.metadata-compaction`**: Defines rules for metadata compaction operations - Schema Definition: @https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json - Optimizes table metadata for improved performance + - Applicable resources: Iceberg table, namespace, catalog - **`system.orphan-file-removal`**: Defines rules for removing orphaned files - Schema Definition: @https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json - Identifies and safely removes files that are no longer referenced by the table metadata + - Applicable resources: Iceberg table, namespace, catalog - **`system.snapshot-expiry`**: Defines rules for snapshot expiration - Schema Definition: @https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json - Controls how long snapshots are retained before removal + - Applicable resources: Iceberg table, namespace, catalog - **Custom policy types**: Can be defined for specific organizational needs (WIP) -- **FGAC (Fine-Grained Access Control) policies**: Row filtering and column masking (WIP) +- **FGAC (Fine-Grained Access Control) policies**: Row filtering, column masking, column hiding (WIP) ### Policy Inheritance The object hierarchy in Polaris is structured as follows: ``` - Catalog - | - Namespace - | - +-----+-----+ - | | - Table View + Catalog + | + Namespace + | + +-----------+----------+ + | | | +Iceberg Iceberg Generic + Table View Table ``` Policies can be attached at any level, and inheritance flows from catalog down to namespace, then to tables and views. Policies can be inheritable or non-inheritable: -- **Inheritable policies**: Apply to the target resource and all its child resources +- **Inheritable policies**: Apply to the target resource and all its applicable child resources - **Non-inheritable policies**: Apply only to the specific target resource The inheritance follows an override mechanism: @@ -155,6 +159,9 @@ PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings For inheritable policies, only one policy of a given type can be attached to a resource. For non-inheritable policies, multiple policies of the same type can be attached. ### Retrieving Applicable Policies +A user can view applicable policies on a resource (e.g., table, namespace, or catalog) as long as they have +read permission on that resource. The permission model may be enhanced in the future when Fine-Grained Access Control +policy is introduced, which will provide more granular control over policy visibility and management. Here is an example to find all policies that apply to a specific resource (including inherited policies): ``` From b7bde019252f49deda47be962fdd523ac6e6019e Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Sat, 17 May 2025 13:37:37 -0700 Subject: [PATCH 5/6] Resolve comments --- site/content/in-dev/unreleased/policy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/in-dev/unreleased/policy.md b/site/content/in-dev/unreleased/policy.md index bdc0734556..32ed71cf93 100644 --- a/site/content/in-dev/unreleased/policy.md +++ b/site/content/in-dev/unreleased/policy.md @@ -70,7 +70,7 @@ Polaris supports several predefined system policy types (prefixed with `system.` ### Policy Inheritance -The object hierarchy in Polaris is structured as follows: +The entity hierarchy in Polaris is structured as follows: ``` Catalog From 3170c364a07e8fe969d245bb23554b24b96c6e3c Mon Sep 17 00:00:00 2001 From: Yufei Gu Date: Mon, 19 May 2025 22:43:43 -0700 Subject: [PATCH 6/6] Resolve comments --- site/content/in-dev/unreleased/policy.md | 40 +++++++++--------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/site/content/in-dev/unreleased/policy.md b/site/content/in-dev/unreleased/policy.md index 32ed71cf93..3f49353884 100644 --- a/site/content/in-dev/unreleased/policy.md +++ b/site/content/in-dev/unreleased/policy.md @@ -45,28 +45,15 @@ predefined conditions. Each policy contains: Polaris supports several predefined system policy types (prefixed with `system.`): -- **`system.data-compaction`**: Defines rules for data compaction operations - - Schema Definition: @https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json - - Controls file compaction to optimize storage and query performance - -- **`system.metadata-compaction`**: Defines rules for metadata compaction operations - - Schema Definition: @https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json - - Optimizes table metadata for improved performance - - Applicable resources: Iceberg table, namespace, catalog +| Policy Type | Purpose | JSON-Schema | Applies To | +|-------------|-------------------------------------------------------|-------------|------------| +| **`system.data-compaction`** | Defines rules for data file compaction operations | [`data-compaction/2025-02-03.json`](https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json) | Iceberg **table**, **namespace**, **catalog** | +| **`system.metadata-compaction`** | Defines rules for metadata file compaction operations | [`metadata-compaction/2025-02-03.json`](https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json) | Iceberg **table**, **namespace**, **catalog** | +| **`system.orphan-file-removal`** | Defines rules for removing orphaned files | [`orphan-file-removal/2025-02-03.json`](https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json) | Iceberg **table**, **namespace**, **catalog** | +| **`system.snapshot-expiry`** | Defines rules for snapshot expiration | [`snapshot-expiry/2025-02-03.json`](https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json) | Iceberg **table**, **namespace**, **catalog** | -- **`system.orphan-file-removal`**: Defines rules for removing orphaned files - - Schema Definition: @https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json - - Identifies and safely removes files that are no longer referenced by the table metadata - - Applicable resources: Iceberg table, namespace, catalog - -- **`system.snapshot-expiry`**: Defines rules for snapshot expiration - - Schema Definition: @https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json - - Controls how long snapshots are retained before removal - - Applicable resources: Iceberg table, namespace, catalog - -- **Custom policy types**: Can be defined for specific organizational needs (WIP) - -- **FGAC (Fine-Grained Access Control) policies**: Row filtering, column masking, column hiding (WIP) +Support for additional predefined system policy types and custom policy type definitions is in progress. +For more details, please refer to the [roadmap](https://github.com/apache/polaris/discussions/1028). ### Policy Inheritance @@ -94,6 +81,9 @@ The inheritance follows an override mechanism: 1. Table-level policies override namespace and catalog policies 2. Namespace-level policies override parent namespace and catalog policies +> **Important:** Because an override completely replaces the same policy type at higher levels, +> **only one instance of a given policy type can be attached to (and therefore affect) a resource**. + ## Working with Policies ### Creating a Policy @@ -110,7 +100,7 @@ POST /polaris/v1/{prefix}/namespaces/{namespace}/policies } ``` -The policy content is validated against a schema specific to its type. Here are a few policy content examples +The policy content is validated against a schema specific to its type. Here are a few policy content examples: - Data Compaction Policy ```json { @@ -156,12 +146,12 @@ PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings } ``` -For inheritable policies, only one policy of a given type can be attached to a resource. For non-inheritable policies, multiple policies of the same type can be attached. +For inheritable policies, only one policy of a given type can be attached to a resource. For non-inheritable policies, +multiple policies of the same type can be attached. ### Retrieving Applicable Policies A user can view applicable policies on a resource (e.g., table, namespace, or catalog) as long as they have -read permission on that resource. The permission model may be enhanced in the future when Fine-Grained Access Control -policy is introduced, which will provide more granular control over policy visibility and management. +read permission on that resource. Here is an example to find all policies that apply to a specific resource (including inherited policies): ```