Skip to content

[SPARK-32156][TESTS][SQL] Refactor two similar test cases from SPARK-31061 in HiveExternalCatalogSuite #28980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-32156][TESTS][SQL] Refactor two similar test cases from SPARK-31061 in HiveExternalCatalogSuite #28980

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e9604a0
temp modify
TJX2014 Jul 2, 2020
17f3928
correct test case and polish
TJX2014 Jul 2, 2020
0fb7829
rename parquetTable to tableDDL to compatible with parquet and hive
TJX2014 Jul 2, 2020
899780e
i => provider
TJX2014 Jul 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 19 additions & 35 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,41 +181,25 @@ class HiveExternalCatalogSuite extends ExternalCatalogSuite {
"INSERT overwrite directory \"fs://localhost/tmp\" select 1 as a"))
}

test("SPARK-31061: alterTable should be able to change table provider") {
test("SPARK-31061: alterTable should be able to change table provider/hive") {
val catalog = newBasicCatalog()
val parquetTable = CatalogTable(
identifier = TableIdentifier("parq_tbl", Some("db1")),
tableType = CatalogTableType.MANAGED,
storage = storageFormat.copy(locationUri = Some(new URI("file:/some/path"))),
Copy link
Contributor Author

@TJX2014 TJX2014 Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file:/some/path seems not very suitable.

schema = new StructType().add("col1", "int").add("col2", "string"),
provider = Some("parquet"))
catalog.createTable(parquetTable, ignoreIfExists = false)

val rawTable = externalCatalog.getTable("db1", "parq_tbl")
assert(rawTable.provider === Some("parquet"))

val fooTable = parquetTable.copy(provider = Some("foo"))
Copy link
Contributor Author

@TJX2014 TJX2014 Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems we should use rawTableget from catalog instead of parquetTable from the beginning.

catalog.alterTable(fooTable)
val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
assert(alteredTable.provider === Some("foo"))
}

test("SPARK-31061: alterTable should be able to change table provider from hive") {
val catalog = newBasicCatalog()
val hiveTable = CatalogTable(
identifier = TableIdentifier("parq_tbl", Some("db1")),
tableType = CatalogTableType.MANAGED,
storage = storageFormat,
schema = new StructType().add("col1", "int").add("col2", "string"),
provider = Some("hive"))
catalog.createTable(hiveTable, ignoreIfExists = false)

val rawTable = externalCatalog.getTable("db1", "parq_tbl")
assert(rawTable.provider === Some("hive"))

val fooTable = rawTable.copy(provider = Some("foo"))
catalog.alterTable(fooTable)
val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
assert(alteredTable.provider === Some("foo"))
Seq("parquet", "hive").foreach( provider => {
val tableDDL = CatalogTable(
identifier = TableIdentifier("parq_tbl", Some("db1")),
tableType = CatalogTableType.MANAGED,
storage = storageFormat,
schema = new StructType().add("col1", "int"),
provider = Some(provider))
catalog.dropTable("db1", "parq_tbl", true, true)
catalog.createTable(tableDDL, ignoreIfExists = false)

val rawTable = externalCatalog.getTable("db1", "parq_tbl")
assert(rawTable.provider === Some(provider))

val fooTable = rawTable.copy(provider = Some("foo"))
catalog.alterTable(fooTable)
val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
assert(alteredTable.provider === Some("foo"))
})
}
}