Duplicate tables with same name in a namespace

[pingtimeout]

Describe the bug

The persistence layer allows the creation of tables with identical name within the same namespace, which should not be possible. Additionally, it seems to be losing some writes.

To Reproduce

• Check out this commit: 4cf08d4
• Run the server using the getting-started docker compose file: docker compose -f getting-started/eclipselink/docker-compose.yml up
• Export the client ID and secrets as environment variables: export CLIENT_ID=root CLIENT_SECRET=s3cr3t
• Run ./gradlew :polaris-benchmarks:gatlingRun

The simulation will create a catalog named C_0, a namespace named N_0 and then will send 50 simultaneous table creation queries for a table named T_1000.

Actual Behavior

The Gatling output shows that all 50 table creation requests returned with an HTTP 200 OK code.

========================================================================================================================
2025-02-27 09:47:34 UTC                                                                               3s elapsed
---- Requests -----------------------------------------------------------------------|---Total---|-----OK----|----KO----
> Global                                                                             |       103 |       103 |         0
> Authenticate                                                                       |        51 |        51 |         0
> Create Catalog                                                                     |         1 |         1 |         0
> Create Namespace                                                                   |         1 |         1 |         0
> Create Table                                                                       |        50 |        50 |         0

A curl command that lists the tables under namespace NS_0 shows that there are 23 tables with that name.

$ curl \
    -s \
    "http://localhost:8181/api/catalog/v1/C_0/namespaces/NS_0/tables" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $TOKEN" \
  | jq '.identifiers[].name' \
  | wc -l
23

This file is the complete output from GET /api/catalog/v1/C_0/namespaces/NS_0/tables

Expected Behavior

Only a single POST /api/catalog/v1/C_0/namespaces/NS_0/tables should succeed. The remaining 49 queries should be rejected with a HTTP 409 error code.

Additionally, given that the 50 POST /api/catalog/v1/C_0/namespaces/NS_0/tables succeeded, but only 23 tables were actually created, it means that the server lost some writes. For this particular case, it is not that big of a deal as this is an invalid situation. But this raises the question whether other writes can be lost under high concurrency.

Additional context

The issue is reproducible fairly consistently. To make it even easier to reproduce, increase log verbosity (e.g. -Dquarkus.log.level=DEBUG -Dquarkus.log.category.\"org.apache.polaris\".level=DEBUG -Dquarkus.log.category.\"org.apache.iceberg..rest\".level=DEBUG -Dquarkus.log.category.\"io.smallrye.config\".level=DEBUG )

System information

No response

[pingtimeout]

Related to #767

[dennishuo]

Thanks for the report, and the handy benchmark tools! Did some digging, looks like there are actually three distinct bugs in play:

1. Failure to configure the db layer with the necessary isolation level (SERIALIZABLE) - in normal prod scenarios this would be configured differently depending on the backend, e.g. if using a "managed Postgres DB", but certainly we should have defense-in-depth - (a) Set the right default in a handy postgres conf file that we add to our getting-started, and (b) Make our EclipseLink initialization automatically set SERIALIZABLE transaction isolation as well
2. Apparently EclipseLink uses a different read connection vs write connection even in the same EntityManager session and normally even if there's a transaction. This means the two connections aren't part of the same transaction, and the reads see a different/dirty view of the database compared to the decisions leading to the writes. https://docs.oracle.com/middleware/1221/toplink/concepts/data_access.htm#CHDJGAJE https://eclipse.dev/eclipselink/documentation/2.4/jpa/extensions/p_transaction.htm
3. There's an unfortunate TODO that fell through the cracks in BasePolarisCatalog in createTableLike and updateTableLike -- historically, it seems the DB layer vacillated between returning a status enum vs throwing an exception, so code at one time relied on exceptions escaping up to cause error return codes. But now with return status enums, the enums need to be translated back into exceptions.

It turns out (1b) requires a bit of extra injection to make it happen: https://eclipse.dev/eclipselink/documentation/2.6/dbws/creating_dbws_services002.htm (I also experimented with trying to unpack the database session at runtime in PolarisEclipseLinkMetaStoreSessionImpl itself and did also get it to work but it's hacky/messy)

Number (2) just requires adding to persistence.xml:

  <property name="eclipselink.transaction.join-existing" value="true"/>

(1) and (2) together are both needed to avoid corruption of the backend database. If these two are fixed, then at least the data model and underlying behavior is correct, but the only problem is erroneous 200 return codes.

(3) Requires handling the returned EntityResult in createTableLike and updateTableLike in BasePolarisCatalog. Fixing this also fixes the erroneous 200 return codes.

[dennishuo]

Confirmed with #1092 and cranking up to concurrency of 100 that I get 99 failures and 1 success:

$ ./gradlew :polaris-benchmarks:gatlingRun
Configuration on demand is an incubating feature.

> Task :polaris-benchmarks:gatlingRun
Simulation IcebergRestSimulation started...

========================================================================================================================
2025-03-02 11:43:58 GMT                                                                               2s elapsed
---- Requests -----------------------------------------------------------------------|---Total---|-----OK----|----KO----
> Global                                                                             |       203 |       104 |        99
> Authenticate                                                                       |       101 |       101 |         0
> Create Catalog                                                                     |         1 |         1 |         0
> Create Namespace                                                                   |         1 |         1 |         0
> Create Table                                                                       |       100 |         1 |        99
---- Errors ------------------------------------------------------------------------------------------------------------
> status.find.in([200, 209], 304), found 409                                                                 99   (100%)

---- Create catalog and namespace using the Iceberg REST API -----------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:         1
---- Create tables using the Iceberg REST API --------------------------------------------------------------------------
[################################################################################################################]  100%
          waiting:         0 / active:         0  / done:       100
========================================================================================================================

Parsing log file(s)...
Parsing log file(s) done in 0s.
Generating reports...

========================================================================================================================
---- Global Information -------------------------------------------------------------|---Total---|-----OK----|----KO----
> request count                                                                      |       203 |       104 |        99
> min response time (ms)                                                             |        11 |        11 |       847
> max response time (ms)                                                             |     1,371 |     1,369 |     1,371
> mean response time (ms)                                                            |       666 |       174 |     1,182
> response time std deviation (ms)                                                   |       521 |       132 |       132
> response time 50th percentile (ms)                                                 |       274 |       168 |     1,186
> response time 75th percentile (ms)                                                 |     1,186 |       210 |     1,289
> response time 95th percentile (ms)                                                 |     1,351 |       261 |     1,362
> response time 99th percentile (ms)                                                 |     1,366 |       275 |     1,371
> mean throughput (rps)                                                              |     67.67 |     34.67 |        33
---- Response Time Distribution ----------------------------------------------------------------------------------------
> OK: t < 800 ms                                                                                            103 (50.74%)
> OK: 800 ms <= t < 1200 ms                                                                                   0     (0%)
> OK: t >= 1200 ms                                                                                            1  (0.49%)
> KO                                                                                                         99 (48.77%)
---- Errors ------------------------------------------------------------------------------------------------------------
> status.find.in([200, 209], 304), found 409                                                                 99   (100%)
========================================================================================================================

Whereas if (1) or (2) aren't fixed, it shows reliable corruption in the backend with duplicates.

If (1) and (2) are fixed but not (3), then the database remains correct with 1 unique table, but there will be erroneous 200 return codes. The 200 responses do return the "correct" metadata JSON (the really committed one, not new attempts to commit), but they should be 409s nonetheless.

[jbonofre]

@pingtimeout are you using the default persistence.xml or a custom one ?

[pingtimeout]

@jbonofre I am using the default one