Skip to content

Conversation

@XN137
Copy link
Contributor

@XN137 XN137 commented Aug 8, 2025

currently one can freely "cast" any PolarisEntity to a more
specific type via their constructors.

this can lead to subtle bugs like we fixed in
a29f800

by adding type checks we discover a few more places where we need to be
more careful about how we construct new or handle existing entities.

note that we can add a check for PolarisEntitySubType in a followup,
but it requires more fixes currently.

@github-project-automation github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Aug 8, 2025
@XN137 XN137 force-pushed the entity-type-check branch 5 times, most recently from 4de57ef to 33c41bf Compare August 8, 2025 16:09
@XN137 XN137 force-pushed the entity-type-check branch 3 times, most recently from c745983 to 214f161 Compare August 25, 2025 07:14
if (rawLeafEntity.getType() == PolarisEntityType.TABLE_LIKE) {
return IcebergTableLikeEntity.of(rawLeafEntity);
}
return null; // could be an external catalog
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without this change the CatalogFederationIT was failing:

getTableEntity: ns1.test_table -> resolvedPath:[entity:name=test_catalog_external;id=6;parentId=0;entityVersion=1;type=CATALOG;subType=NULL_SUBTYPE;internalProperties={catalogType=EXTERNAL, storage_configuration_info={"@type":"FileStorageConfigurationInfo","allowedLocations":["file:///tmp/warehouse"],"storageType":"FILE","fileIoImplClassName":"org.apache.iceberg.hadoop.HadoopFileIO"}, connection_configuration_info={"uri":"http://localhost:42705/api/catalog","authenticationParameters":{"tokenUri":"http://localhost:42705/api/catalog/v1/oauth/tokens","clientId":"71667f9b548ce462","clientSecretReference":{"urn":"urn:polaris-secret:unsafe-in-memory:6:0","referencePayload":{"ciphertext-hash":"adf96dbbdf0d7b85a4940ff1e5bd912526fab136e9b66896efcd641be8bd3c6c","encryption-key":"hDB047wH0cue+kMm6v28HhJI9cK6JOs63SWqzR39V+o="}},"scopes":["PRINCIPAL_ROLE:ALL"],"authenticationTypeCode":1},"remoteCatalogName":"test_catalog_local","connectionTypeCode":1}};grantRecordsAsGrantee:[];grantRecordsAsSecurable:[PolarisGrantRec{securableCatalogId=0, securableId=6, granteeCatalogId=6, granteeId=7, privilegeCode=2}, PolarisGrantRec{securableCatalogId=0, securableId=6, granteeCatalogId=6, granteeId=7, privilegeCode=21}, PolarisGrantRec{securableCatalogId=0, securableId=6, granteeCatalogId=6, granteeId=7, privilegeCode=31}]] -> name=test_catalog_external;id=6;parentId=0;entityVersion=1;type=CATALOG;subType=NULL_SUBTYPE;internalProperties={catalogType=EXTERNAL, storage_configuration_info={"@type":"FileStorageConfigurationInfo","allowedLocations":["file:///tmp/warehouse"],"storageType":"FILE","fileIoImplClassName":"org.apache.iceberg.hadoop.HadoopFileIO"}, connection_configuration_info={"uri":"http://localhost:42705/api/catalog","authenticationParameters":{"tokenUri":"http://localhost:42705/api/catalog/v1/oauth/tokens","clientId":"71667f9b548ce462","clientSecretReference":{"urn":"urn:polaris-secret:unsafe-in-memory:6:0","referencePayload":{"ciphertext-hash":"adf96dbbdf0d7b85a4940ff1e5bd912526fab136e9b66896efcd641be8bd3c6c","encryption-key":"hDB047wH0cue+kMm6v28HhJI9cK6JOs63SWqzR39V+o="}},"scopes":["PRINCIPAL_ROLE:ALL"],"authenticationTypeCode":1},"remoteCatalogName":"test_catalog_local","connectionTypeCode":1}}
2025-08-25 08:43:48,888 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-1) Handling runtimeException Invalid entity type: CATALOG
2025-08-25 08:43:48,888 INFO  [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-1) Full RuntimeException: java.lang.IllegalStateException: Invalid entity type: CATALOG
	at com.google.common.base.Preconditions.checkState(Preconditions.java:603)
	at org.apache.polaris.core.entity.table.TableLikeEntity.<init>(TableLikeEntity.java:41)
	at org.apache.polaris.core.entity.table.IcebergTableLikeEntity.<init>(IcebergTableLikeEntity.java:49)
	at org.apache.polaris.core.entity.table.IcebergTableLikeEntity.of(IcebergTableLikeEntity.java:54)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.getTableEntity(IcebergCatalogHandler.java:578)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.loadTableIfStale(IcebergCatalogHandler.java:603)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$loadTable$10(IcebergCatalogAdapter.java:431)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:193)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.loadTable(IcebergCatalogAdapter.java:422)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.loadTable(Unknown Source)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder how the entity path ends up pointing to a CATALOG when it's a loadTable call 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idk how external catalog integration works under the hood but in the first output line (that i had added) we can see that the PolarisResolvedPathWrapper ends up pointing to the external catalog entity (which is also the leaf element).

so i assumed for external catalogs in general we dont have any actual entity representing the table? but maybe this is just an artifact of an incorrectly written test idk.

the return value of getTableEntity is only used for supporting ETags not for loading the table:

if (ifNoneMatch != null) {
// Perform freshness-aware table loading if caller specified ifNoneMatch.
IcebergTableLikeEntity tableEntity = getTableEntity(tableIdentifier);
if (tableEntity == null || tableEntity.getMetadataLocation() == null) {
LOGGER
.atWarn()
.addKeyValue("tableIdentifier", tableIdentifier)
.addKeyValue("tableEntity", tableEntity)
.log("Failed to getMetadataLocation to generate ETag when loading table");
} else {
// TODO: Refactor null-checking into the helper method once we create a more canonical
// interface for associate etags with entities.
String tableEntityTag =
IcebergHttpUtil.generateETagForMetadataFileLocation(tableEntity.getMetadataLocation());
if (ifNoneMatch.anyMatch(tableEntityTag)) {
return Optional.empty();
}
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dennishuo : do you have any insight to share?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be clear: this conversation is interesting, but the changes in this PR still make sense from my POV 🙂

@XN137 XN137 force-pushed the entity-type-check branch from 214f161 to db464b6 Compare August 26, 2025 14:55
@XN137 XN137 changed the title Add type-check to PolarisEntity ctors Add type-check to PolarisEntity subclass ctors Aug 26, 2025
@XN137 XN137 marked this pull request as ready for review August 26, 2025 15:11
@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 26, 2025
Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refactoring LGTM 👍

currently one can freely "cast" any `PolarisEntity` to a more
specific type via their constructors.

this can lead to subtle bugs like we fixed in
a29f800

by adding type checks we discover a few more places where we need to be
more careful about how we construct new or handle existing entities.

note that we can add a check for `PolarisEntitySubType` in a followup,
but it requires more fixes currently.
@XN137 XN137 force-pushed the entity-type-check branch from e2c507b to b178a41 Compare September 1, 2025 03:31
@dimas-b dimas-b merged commit ac31963 into apache:main Sep 2, 2025
12 checks passed
@github-project-automation github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 2, 2025
@XN137 XN137 deleted the entity-type-check branch September 3, 2025 04:45
snazy added a commit to snazy/polaris that referenced this pull request Nov 20, 2025
* Integration tests for Catalog Federation (apache#2344)

Adds a Junit5 integration test for catalog federation.

* Fix merge conflict in CatalogFederationIntegrationTest (apache#2420)

apache#2344 added a new test for catalog federation, but it looks like an undetected conflict with concurrent changes related to authentication have broken the test in main.

* chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6.1755674729 (apache#2416)

* 2334 (apache#2427)

* Fix TableIdentifier in TaskFileIOSupplier (apache#2304)

we cant just convert a `TaskEntity` to a `IcebergTableLikeEntity` as the
`getTableIdentifier()` method will not return a correct value by using
the name of the task and its parent namespace (which is empty?).

task handlers instead need to pass in the `TableIdentifier` that they
already inferred via `TaskEntity.readData`.

* Fix NPE in CreateCatalog (apache#2435)

* Doc fix: Access control page update (apache#2424)

* 2418

* 2418

* fix(deps): update dependency software.amazon.awssdk:bom to v2.32.29 (apache#2443)

* Optimize PolicyCatalog.listPolicies (apache#2370)

this is a follow-up to apache#2290

the optimization is to use `listEntities` instead of `loadEntities` when
there is `policyType` filter to apply

* Add PolarisDiagnostics field to BaseMetaStoreManager (apache#2381)

* Add PolarisDiagnostics field to BaseMetaStoreManager

the ultimate goal is removing the `PolarisCallContext` parameter from every
`PolarisMetaStoreManager` interface method, so we make steps towards
reducing its usage first.

* Add feature flag to disallow custom S3 endpoints (apache#2442)

* Add new realm-level flag: `ALLOW_SETTING_S3_ENDPOINTS` (default: true)

* Enforce in `PolarisServiceImpl.validateStorageConfig()`

Fixes apache#2436

* Deprecate ActiveRolesProvider for removal (apache#2404)

* Client: fix openapi verbose output, remove doc generate, and skip test generations (apache#2439)

* Fix various issue in client code generation

* Use logger instead of print

* Add back exclude on __pycache__ as CI is not via Makefile

* Add back exclude on __pycache__ as CI is not via Makefile

* Add user principal tag in metrics (apache#2445)

* Added API change to enable tag

* Added test

* Added production readiness check

* fix(deps): update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.36.0 (apache#2454)

* fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.56.0 (apache#2447)

* fix(deps): update dependency gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext to v1.3 (apache#2428)

* Build: Make jandex dependency used for index generation managed (apache#2431)

Also allows specifying the jandex index version for the build.

This is a preparation step contributing to apache#2204, once a jandex fix for reproducible builds is available.

Co-authored-by: Alexandre Dutra <[email protected]>

* Built: improve reproducible archive files (apache#2432)

As part of the effort for apache#2204, this change fixes a few aspects around reproducible builds:

Some Gradle projects produce archive files, but don't get the necessary Gradle archive-tasks settings applied: one not-published project but also the tarball&zip of the distribution. This change moves the logic to the new build-plugin `polaris-reproducible`.

Another change is to have some Quarkus generated jar files adhere to the same conventions, which are constant timestamps for the zip entries and a deterministic order of the entries. That's sadly not a full fix, as the classes that are generated or instumented by Quarkus differ in each build.

Contributes to apache#2204

* Remove commons-lang3 dependency (apache#2456)

outside of tests we can replace the functionality with jdk11 and guava.
also stop using `org.assertj.core.util` as its a non-public api.

* add refresh credentials property to loadTableResult (apache#2341)

* add refresh credentials property to loadTableResult

* IcebergCatalogAdapterTest: Added test to ensure refresh credentials endpoint is included

* delegate refresh credential endpoint configuration to storage integration

* GCP: Add refresh credential properties

* fix(deps): update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.37.0 (apache#2458)

* Add Delegator to all API Implementations (apache#2434)

Per the Dev ML, implements the Delegator pattern to add Events instrumentation to all Polaris APIs.

* Prefer java.util.Base64 over commons-codec (apache#2463)

`java.util.Base64` is available since java8 and we are already using it
in a few other spots.

in a follow-up we might be able to get rid of our `commons-codec` dependency
completely.

* Service: Move tests to the right package (apache#2469)

* Update versions in runtime LICENSE and NOTICE (apache#2468)

* fix(deps): update dependency com.adobe.testing:s3mock-testcontainers to v4.8.0 (apache#2475)

* fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.1.0 (apache#2476)

* Service: Remove hadoop-common from polaris-runtime-service (apache#2462)

* Service: Always validate allowed locations from Storage Config (apache#2473)

* Add Community Sync Meeting 20250828 (apache#2477)

* Update dependency software.amazon.awssdk:bom to v2.33.0 (apache#2483)

* Remove PolarisCallContext.getDiagServices (apache#2415)

* Remove PolarisCallContext.getDiagServices usage

* Remove diagnostics from PolarisCallContext

* Feature: Expose resetCredentials via a new reset api to allow root user to reset credentials for an existing principal with custom values  (apache#2197)

* Add type-check to PolarisEntity subclass ctors (apache#2302)

currently one can freely "cast" any `PolarisEntity` to a more
specific type via their constructors.

this can lead to subtle bugs like we fixed in
a29f800

by adding type checks we discover a few more places where we need to be
more careful about how we construct new or handle existing entities.

note that we can add a check for `PolarisEntitySubType` in a followup,
but it requires more fixes currently.

* Fix CI (apache#2489)

Fix undetected merge conflict after apache#2197 + apache#2415 + apache#2434

* Use local diagnostics in TransactionWorkspaceMetaStoreManager

* Add resetCredentials to PolarisPrincipalsEventServiceDelegator

* Core: Prevent AIOOBE for negative codes in PolarisEntityType, PolarisPrivilege, ReturnStatus (apache#2490)

* feat(idgen): Start Implementation of NoSQL with the ID Generation Framework (apache#2131)

Create an ID Generation Framework.

Related to apache#650 & apache#844

Co-authored-by: Robert Stupp <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>

* perf(refactor): optimizing JdbcBasePersistenceImpl.listEntities (apache#2465)

- Reduced Column Selection: Only 6 columns instead of 16

- Eliminated Object Creation Overhead: Direct conversion to EntityNameLookupRecord without intermediate PolarisBaseEntity

* Add Polaris Events to Persistence (apache#1844)

* AWS CloudWatch Event Sink Implementation (apache#1965)

* Fix failing CI (apache#2498)

* Update actions/stale digest to 3a9db7e (apache#2499)

* Core: Prevent AIOOBE for negative policy codes in PredefinedPolicyType (apache#2486)

* Service: Add location tests for views (apache#2496)

* Update docker.io/jaegertracing/all-in-one Docker tag to v1.73.0 (apache#2500)

* Update dependency io.netty:netty-codec-http2 to v4.2.5.Final (apache#2495)

* Update actions/setup-python action to v6 (apache#2502)

* Update the Release Guide about the Helm Chart package (apache#2179)

* Update the Release Guide about the Helm Chart package

* Update release-guide.md

Co-authored-by: Pierre Laporte <[email protected]>

* Add missing commit message

* Whitespace

* Use Helm GPG plugin to sign the Helm chart

* Fix directories during Helm chart copy to SVN

* Add Helm index to SVN

* Use long name for svn checkout

* Ensure the Helm index is updated after the chart is moved to SVN dist release

* Do not publish any Docker image before the vote succeeds

* Typos

* Revert "Do not publish any Docker image before the vote succeeds"

This reverts commit 5617e65.

* Don't mention Helm values.yaml in the release guide as it doesn't contain version details

---------

Co-authored-by: Pierre Laporte <[email protected]>

* Update dependency com.azure:azure-sdk-bom to v1.2.38 (apache#2503)

* Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.23-6.1756793420 (apache#2504)

* Remove commons-codec dependency (apache#2474)

follow-up to f8ad77a

we can simply use guava instead and eliminate the extra dependency

* CLI: Remove SCRIPT_DIR and default config location to user home (apache#2448)

* Remove readInternalProperties helpers (apache#2506)

the functionality is already provided by the `PrincipalEntity`

* Add Events for Generic Table APIs (apache#2481)


This PR adds the Events instrumentation for the Generic Tables Service APIs, surrounding the default delegated call to the business logic APIs.

* Disable custom namespace locations (apache#2422)

When we create a namespace or alter its location, we must confirm that this location is within the parent location. This PR introduces introduces a check similar to the one we have for tables, where custom locations are prohibited by default. This functionality is gated behind a new behavior change flag `ALLOW_NAMESPACE_CUSTOM_LOCATION`. In addition to allowing us to revert to the old behavior, this flag allows some tests relying on arbitrarily-located namespaces to pass (such as those from upstream Iceberg).

Fixes: apache#2417

* fix for IcebergAllowedLocationTest (apache#2511)

* Remove unused config from SparkSessionBuilder (apache#2512)

Tests pass without it.

* Add Events for Policy Service APIs (apache#2479)

* Remove PolarisTestMetaStoreManager.jsonNode helper (apache#2513)

* Update dependency software.amazon.awssdk:bom to v2.33.4 (apache#2517)

* Update dependency com.nimbusds:nimbus-jose-jwt to v10.5 (apache#2514)

* Update dependency io.opentelemetry:opentelemetry-bom to v1.54.0 (apache#2515)

* Update dependency io.micrometer:micrometer-bom to v1.15.4 (apache#2519)

* Port missed OSS change

* NoSQL: adopt to updated test packages

* NoSQL: adapt to removed PolarisDiagnostics param

* NoSQL: fix libs.versions.toml

* NoSQL: include jandex plugin related changes from OSS

* NoSQL: changes for delete/set principal client-ID+secret

* Last merged commit c6176dc

---------

Co-authored-by: Pooja Nilangekar <[email protected]>
Co-authored-by: Eric Maynard <[email protected]>
Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: Yong Zheng <[email protected]>
Co-authored-by: Christopher Lambert <[email protected]>
Co-authored-by: Honah (Jonas) J. <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>
Co-authored-by: Alexandre Dutra <[email protected]>
Co-authored-by: fivetran-kostaszoumpatianos <[email protected]>
Co-authored-by: Jason <[email protected]>
Co-authored-by: Adnan Hemani <[email protected]>
Co-authored-by: Yufei Gu <[email protected]>
Co-authored-by: JB Onofré <[email protected]>
Co-authored-by: fivetran-arunsuri <[email protected]>
Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com>
Co-authored-by: Artur Rakhmatulin <[email protected]>
Co-authored-by: Pierre Laporte <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants