Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions helm/polaris/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ A Helm chart for Apache Polaris (incubating).

### Optional

When using a custom `persistence.xml`, a Kubernetes Secret must be created for it. Below is a sample command:
When using EclipseLink backed metastore a custom `persistence.xml` is required, a Kubernetes Secret must be created for it. Below is a sample command:
```bash
kubectl create secret generic polaris-secret -n polaris --from-file=persistence.xml
```
Expand All @@ -67,7 +67,7 @@ helm unittest helm/polaris
The below instructions assume Kind and Helm are installed.

Simply run the `run.sh` script from the Polaris repo root, making sure to specify the
`--eclipse-link-deps` option:
`--eclipse-link-deps` if using EclipseLink backed metastore, option:

```bash
./run.sh
Expand Down Expand Up @@ -186,8 +186,8 @@ kubectl delete namespace polaris

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
Key | Type | Default | Description |
|-----|------|-----|-------------|
| advancedConfig | object | `{}` | Advanced configuration. You can pass here any valid Polaris or Quarkus configuration property. Any property that is defined here takes precedence over all the other configuration values generated by this chart. Properties can be passed "flattened" or as nested YAML objects (see examples below). Note: values should be strings; avoid using numbers, booleans, or other types. |
| affinity | object | `{}` | Affinity and anti-affinity for polaris pods. See https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity. |
| authentication | object | `{"authenticator":{"type":"default"},"tokenBroker":{"maxTokenGeneration":"PT1H","secret":{"name":null,"privateKey":"private.pem","publicKey":"public.pem","secretKey":"secret"},"type":"rsa-key-pair"},"tokenService":{"type":"default"}}` | Polaris authentication configuration. |
Expand Down Expand Up @@ -285,7 +285,7 @@ kubectl delete namespace polaris
| persistence.eclipseLink.secret | object | `{"key":"persistence.xml","name":null}` | The secret name to pull persistence.xml from. |
| persistence.eclipseLink.secret.key | string | `"persistence.xml"` | The key in the secret to pull persistence.xml from. |
| persistence.eclipseLink.secret.name | string | `nil` | The name of the secret to pull persistence.xml from. If not provided, the default built-in persistence.xml will be used. This is probably not what you want. |
| persistence.type | string | `"eclipse-link"` | The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link. |
| persistence.type | string | `"relational-jdbc"` | Three built-in types are available: "relational-jdbc", "in-memory", "eclipse-link". The in-memory type is not recommended for production use. The eclipse-link type is deprecated and will be unsupported in a future release. |
| podAnnotations | object | `{}` | Annotations to apply to polaris pods. |
| podLabels | object | `{}` | Additional Labels to apply to polaris pods. |
| podSecurityContext | object | `{"fsGroup":10001,"seccompProfile":{"type":"RuntimeDefault"}}` | Security context for the polaris pod. See https://kubernetes.io/docs/tasks/configure-pod-container/security-context/. |
Expand Down Expand Up @@ -343,4 +343,4 @@ kubectl delete namespace polaris
| tracing.attributes | object | `{}` | Resource attributes to identify the polaris service among other tracing sources. See https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service. If left empty, traces will be attached to a service named "Apache Polaris"; to change this, provide a service.name attribute here. |
| tracing.enabled | bool | `false` | Specifies whether tracing for the polaris server should be enabled. |
| tracing.endpoint | string | `"http://otlp-collector:4317"` | The collector endpoint URL to connect to (required). The endpoint URL must have either the http:// or the https:// scheme. The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317). See https://quarkus.io/guides/opentelemetry for more information. |
| tracing.sample | string | `"1.0d"` | Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled. Note: avoid entering numbers here, always prefer a string representation of the ratio. |
| tracing.sample | string | `"1.0d"` | Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled. Note: avoid entering numbers here, always prefer a string representation of the ratio. |
2 changes: 1 addition & 1 deletion quarkus/admin/src/main/resources/application.properties
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ quarkus.container-image.additional-tags=latest
# ---- Runtime Configuration ----
# Below are default values for properties that can be changed in runtime.

polaris.persistence.type=eclipse-link
polaris.persistence.type=relational-jdbc

quarkus.arc.ignored-split-packages=\
org.apache.polaris.admintool.config,\
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,10 @@ public Map<String, String> start() {
context.containerNetworkId().ifPresent(postgres::withNetworkMode);
postgres.start();
return Map.of(
"polaris.persistence.eclipselink.configuration-file", createPersistenceXml().toString());
"polaris.persistence.type",
"eclipse-link",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because all tests currently run against the EclipseLink implementation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we deprecate EclipseLink, why not purge it from tests too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can keep it for now (even deprecated) and remove when we remove eclipselink.

Copy link
Contributor

@dimas-b dimas-b May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a separate test profile for EclipseLink then and run maybe a sub-set of tests. How about the bootstrap test in the admin tool and the REST API test in the quarkus server?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: removing this change actually makes all tests use JDBC, as far as I understand 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be best to keep doc changes in this PR and make application.properties and test changes in another PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a separate test profile for EclipseLink then and run maybe a sub-set of tests. How about the bootstrap test in the admin tool and the REST API test in the quarkus server?

This profile is just used for eclipse link, the jdbc profile and lifecycle manager is is present is quarkus/test-commons. The reason why i kept both together is i we are deprecating eclipse link via doc and not via code, so hence updating default is in the same PR. Please Let me know your thoughts considering above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. thx for the clarification!

"polaris.persistence.eclipselink.configuration-file",
createPersistenceXml().toString());
}

@Override
Expand Down
11 changes: 3 additions & 8 deletions site/content/in-dev/unreleased/admin-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,18 +74,13 @@ The Polaris Admin Tool must be executed with the same configuration as the Polar
configuration can be done via environment variables or system properties.

At a minimum, it is necessary to configure the Polaris Admin Tool to connect to the same database
used by the Polaris server. This can be done by setting the following system properties:

```shell
java \
-Dpolaris.persistence.eclipselink.configuration-file=/path/to/persistence.xml \
-Dpolaris.persistence.eclipselink.persistence-unit=polaris \
-jar quarkus/admin/build/polaris-quarkus-admin-*-runner.jar
```
used by the Polaris server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do users need to do any action here to make this happen? If not, would it be better to remove this line altogether?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but the immediate line below re-directs the user to move to metastore.md page to configure the metastore, so this was kind a redundant. Hence removed.


See the [metastore documentation]({{% ref "metastores" %}}) for more information on configuring the
database connection.

Note: Polaris will always create schema 'polaris_schema' during bootstrap under the configured database.

## Bootstrapping Realms and Principal Credentials

The `bootstrap` command is used to bootstrap realms and create the necessary principal credentials
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,34 +135,32 @@ default, Polaris uses an in-memory metastore.
> The default in-memory metastore is not suitable for production use, as it will lose all data
> when the server is restarted; it is also unusable when multiple Polaris replicas are used.

To use a durable metastore, you need to switch to the EclipseLink metastore, and provide your own
`persistence.xml` file. This file contains details of the database used for metastore management and
the connection settings. For more information, refer to the [metastore documentation]({{% ref
"metastores" %}}).
To enable a durable metastore, configure your system to use the Relational JDBC-backed metastore.
This implementation leverages Quarkus for datasource management and supports configuration through
environment variables or JVM -D flags at startup. For more information, refer to the [Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file).

Then, configure Polaris to use your metastore by setting the following properties:
Configure the metastore by setting the following ENV variables:

```properties
polaris.persistence.type=eclipse-link
polaris.persistence.eclipselink.configuration-file=/path/to/persistence.xml
polaris.persistence.eclipselink.persistence-unit=polaris
```
POLARIS_PERSISTENCE_TYPE=relational-jdbc

Where:
QUARKUS_DATASOURCE_DB_KIND=postgresql
QUARKUS_DATASOURCE_USERNAME=<your-username>
QUARKUS_DATASOURCE_PASSWORD=<your-password>
QUARKUS_DATASOURCE_JDBC_URL=<jdbc-url-of-postgres>
```
Comment on lines +142 to +151
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both env variable and Quarkus application.properties work. Can we be consistent about the configuration in the doc as the metastores.md use the application.properties way? I think we can mention the application.properties way here while refer to page https://polaris.apache.org/in-dev/unreleased/configuration/ if users prefer env variable. Thoughts?

Copy link
Contributor

@dimas-b dimas-b May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I support consistency.

As to using application.properties, as I commented in another thread, they are relative to current dir and as such prone to errors, IMHO.

If we want to go with config files, I propose to revive @jbonofre PR on that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with either way. Hi @dimas-b , can you share JB's PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also supportive on JB's PR. It provides much better UX in terms of how users config Polaris server and admin tool. User can make changes to config/application.prooperties without searching for Polaris doc how to config things. We can even put certain common used configures in it, and comment them out by default, and users can uncomment them when needed.


- `polaris.persistence.type` indicates that we are using the EclipseLink metastore.
- `polaris.persistence.eclipselink.configuration-file` is the path to the `persistence.xml` file.
- `polaris.persistence.eclipselink.persistence-unit` is the name of the persistence unit to use (in
case the configuration file has many persistence units).

Typically, in Kubernetes, you would define the `persistence.xml` file as a `ConfigMap` and set the
`polaris.persistence.eclipselink.configuration-file` property to the path of the mounted file in
the container.
The relational JDBC metastore is a Quarkus-managed datasource and only supports Postgres and H2 as of now.
Please refer to the documentation here:
[Configure data sources in Quarkus](https://quarkus.io/guides/datasource)

> [!IMPORTANT]
> Be sure to secure your metastore backend since it will be storing sensitive data and catalog
> metadata.

Note: Polaris will always create schema 'polaris_schema' during bootstrap under the configured database.

### Bootstrapping

Before using Polaris, you must **bootstrap** the metastore. This is a manual operation that must be
Expand Down
26 changes: 24 additions & 2 deletions site/content/in-dev/unreleased/metastores.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,27 @@ In order to add other JDBC drivers, you have to build Polaris using the `eclipse

## Polaris Server Configuration

### Relational JDBC
This implementation leverages Quarkus for datasource management and supports configuration through
environment variables or JVM -D flags at startup. For more information, refer to the [Quarkus configuration reference](https://quarkus.io/guides/config-reference#env-file).


```
POLARIS_PERSISTENCE_TYPE=relational-jdbc

QUARKUS_DATASOURCE_DB_KIND=postgresql
QUARKUS_DATASOURCE_USERNAME=<your-username>
QUARKUS_DATASOURCE_PASSWORD=<your-password>
QUARKUS_DATASOURCE_JDBC_URL=<jdbc-url-of-postgres>
```

The Relational JDBC metastore currently relies on a Quarkus-managed datasource and supports only PostgreSQL and H2 databases. This limitation is similar to that of EclipseLink, primarily due to underlying schema differences. At this time, official documentation is provided exclusively for usage with PostgreSQL.
Please refer to the documentation here:
[Configure data sources in Quarkus](https://quarkus.io/guides/datasource)

### EclipseLink
> [!IMPORTANT] Eclipse link is deprecated, its recommend to use Relational JDBC as persistence instead.

Configure the `polaris.persistence` section in your Polaris configuration file
(`application.properties`) as follows:

Expand All @@ -58,7 +79,7 @@ Polaris creates and connects to a separate database for each realm. Specifically

A single `persistence.xml` can describe multiple [persistence units](https://eclipse.dev/eclipselink/documentation/4.0/concepts/concepts.html#APPDEV001). For example, with both a `polaris-dev` and `polaris` persistence unit defined, you could use a single `persistence.xml` to easily switch between development and production databases. Use the `persistence-unit` option in the Polaris server configuration to easily switch between persistence units.

### Using H2
#### Using H2

> [!IMPORTANT] H2 is an in-memory database and is not suitable for production!

Expand Down Expand Up @@ -101,7 +122,7 @@ java -Dpolaris.persistence.type=eclipse-link \
-jar quarkus/server/build/quarkus-app/quarkus-run.jar
```

### Using Postgres
#### Using Postgres

PostgreSQL is included by default in the Polaris server distribution.

Expand Down Expand Up @@ -130,3 +151,4 @@ The following shows a sample configuration for integrating Polaris with Postgres
</properties>
</persistence-unit>
```

Loading