[SPARK-39506][SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible

[amaliujia]

What changes were proposed in this pull request?

1. add 3l namespace test for CacheTable, isCache.
2. make UncacheTable be 3l namespace compatible.
3. add new API setCurrentCatalog, currentCatalog, listCatalogs which are useful for 3l namespace.

Why are the changes needed?

This is a part of effort for 3l namespace work.

Does this PR introduce any user-facing change?

Yes. There are 3 new API added in this PR (setCurrentCatalog, currentCatalog, listCatalogs) and unCacheTable API will support 3l namespace.

How was this patch tested?

UT

[amaliujia]

R: @cloud-fan

[AmplabJenkins]

Can one of the admins verify this patch?

[cloud-fan]

if it's simply a string, shall we just let listCatalogs return Dataset[String]?

[amaliujia]

We might design the class like this

case class CatalogMetadata(
    name: String,
    description: String,
    owner: Option[String] = None,
    catalogType: Option[CatalogType] = None,
    createdAt: Option[Long] = None,
    createdBy: Option[String] = None,
    updatedAt: Option[Long] = None,
    updatedBy: Option[String] = None
)

However, I am not sure if there are extra information we can fetch for a catalog. Do you know if we have such rich metadata for a catalog in Spark?

[amaliujia]

If Spark only knows the catalog name, then probably Dataset[String] would be fine.

[cloud-fan]

ok let's keep it then, and add a description field so that it doesn't look so weird.

About the naming, how about just Catalog? to match Database and Table in this file.

[amaliujia]

Sounds good for keeping the class and additionally adding a description field.

Regrading renaming to Catalog, it will have a naming conflict with

spark/sql/core/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala

Line 33 in 59eee98

abstract class Catalog {

[cloud-fan]

ah I see!

[amaliujia]

Also I am seeing this

[info] spark-streaming-kinesis-asl: mimaPreviousArtifacts not set, not analyzing binary compatibility
[error] spark-sql: Failed binary compatibility check against org.apache.spark:spark-sql_2.12:3.2.0! Found 3 potential problems (filtered 602)
[error]  * abstract method currentCatalog()java.lang.String in class org.apache.spark.sql.catalog.Catalog is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog")
[error]  * abstract method setCurrentCatalog(java.lang.String)Unit in class org.apache.spark.sql.catalog.Catalog is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog")
[error]  * abstract method listCatalogs()org.apache.spark.sql.Dataset in class org.apache.spark.sql.catalog.Catalog is present only in current version
[error]    filter with: ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")

Any idea how to make the new API pass such API compatibility check?

[HyukjinKwon]

Adding @zhengruifeng FYI

[cloud-fan]

not related to this PR, but we should figure out why spark_catalog is missed here.

[cloud-fan]

The last commit only updates a comment, I'm merging it to master, thanks!

[amaliujia]

@cloud-fan thank you!