./regtests/run_spark_sql.sh broken

[flyrain]

Describe the bug

The script failed to launch spark sql shell.

To Reproduce

1. ./gradlew run
2. Run Spark sql in another terminal ./regtest/run_spark_sql.sh

Actual Behavior

{"error":{"message":"Unsupported storage type: FILE","type":"IllegalArgumentException","code":400}}HTTP/1.1 404 Not Found
content-length: 93
Content-Type: application/json

{"error":{"message":"Catalog not found: manual_spark","type":"NotFoundException","code":404}}HTTP/1.1 404 Not Found
content-length: 157
Content-Type: application/json

Exception in thread "main" org.apache.iceberg.exceptions.RESTException: Unable to process: Unable to find warehouse manual_spark
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:224)
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:188)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262)
	at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358)
	at org.apache.iceberg.rest.RESTSessionCatalog.fetchConfig(RESTSessionCatalog.java:1045)
	at org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:254)
	at org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:78)
	at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:274)
	at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:328)
	at org.apache.iceberg.spark.SparkCatalog.buildIcebergCatalog(SparkCatalog.java:154)
	at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:753)
	at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:65)

Expected Behavior

Launch a spark sql shell

Additional context

This should be caused by #1532.

[flyrain]

Hi @snazy, would you mind taking a look at this issue?

[snazy]

Can you explain the actual issue - CI is passing, so 🤷

[flyrain]

There are reproducing steps, the expected behavior and the actual behavior in the issue body. Can you take a look?

[dimas-b]

Unsupported storage type: FILE - it was removed from the default list of supported storage types in #1532

@flyrain : why would one expect regtest/run_spark_sql.sh to pass with the default server config now?

[flyrain]

This script has been there from the day one of the project, which allow users to test Polaris + Spark sql locally without a docker env.

[dimas-b]

That is a fair point, but then the server has to be configured to allow local files for storage (which is no longer allowed by default).

[dimas-b]

@flyrain : do you mean to say that ./gradlew run should automatically enable the FILE storage type?

[flyrain]

That's correct. I also think it's fine to do so, as no one will use ./gradlew run in production.

[dimas-b]

Well, in this case, I believe this issue ought to be re-titled and description updated to reflect the expected behaviour more accurately.