pyiceberg always return false for catalog.table_exists when using Snowflake managed Polaris service

[djouallah]

using polaris managed offering and pyiceberg 0.7, catalog.table_exists will return false even if the table exist

[eric-maynard]

Hi @djouallah, this project is for OSS Polaris and not any vendor-managed deployment of Polaris. Having said that, I'm interested in taking a look -- can you share your code? I was not able to reproduce this.

I created a config file pyiceberg.json:

{
  "type": "rest",
  "base_uri": "http://localhost:8181/api/catalog",
  "warehouse": "issue_96_catalog",
  "credential": "XXXX:YYYY",
  "scope": "PRINCIPAL_ROLE:ALL"
}

I used the CLI to create the catalog, and used another engine to create a namespace and table in the catalog.

I then ran this Python script using PyIceberg:

from pyiceberg.catalog.rest import RestCatalog
import json

with open('pyiceberg.json', 'r') as f:
    config = json.load(f)

catalog = RestCatalog(
    name='issue_96_catalog',
    uri=config['base_uri'],
    warehouse=config['warehouse'],
    credential=config['credential'],
    scope=config['scope']
)

print(catalog.list_namespaces())
print(catalog.list_tables('ns'))
print(catalog.table_exists('ns.t1'))

I saw that the table_exists call behaved as expected:

$ python -m test
[('ns',)]
[('ns', 't1')]
True

[djouallah]

all good, I will open a ticket i guess.

[kevinjqliu]

Chiming into this, PyIceberg currently has an issue when parsing the table identifier with the "catalog name".

How did you initialize the catalog object in PyIceberg? The catalog name must match the one in the config.
For example, the config value ("warehouse": "issue_96_catalog") in the config file must matches the RestCatalog's parameter (name='issue_96_catalog').

[djouallah]

case 'polaris':
     catalog = load_catalog(
          'default',
           uri='https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxx.snowflakecomputing.com/polaris/api/catalog',
           warehouse='s3',
           scope = 'PRINCIPAL_ROLE:no' ,
           credential= userdata.get("polaris_key")
        )

[kevinjqliu]

In PyIceberg, the catalog is named default. I'm not familiar with the Polaris UI, but I assume it's named s3.

If you rename the PyIceberg catalog to s3 and retry the code above, it might work.

[djouallah]

no luck :(

[kevinjqliu]

Weird, I'll try to repro

[TomerHeber]

To check table existence pyiceberg calls "HEAD" via the API.

In OSS Polaris it currently works as expected.

If I had to guess, the managed Polaris may not be listening to HEAD requests and therefore might be returning 404 which translates to False.