[SPARK-19229] [SQL] Disallow Creating Hive Source Tables when Hive Support is Not Enabled #16587
Conversation
| |Hive support is required to insert into the following tables: | ||
| |${s.catalogTable.identifier} | ||
| """.stripMargin) | ||
|
|
There was a problem hiding this comment.
This is not necessary when we blocking users to create Hive source tables. When the table is data source tables, SimpleCatalogRelation will be replaced by LogicalRelation in the rule FindDataSourceTable.
|
|
||
| test("alter table: set serde partition") { | ||
| // TODO: move this test to HiveDDLSuite.scala | ||
| ignore("alter table: set serde partition") { |
There was a problem hiding this comment.
These two test cases should be moved to HiveDDLSuite. However, it is ugly to copy the codes. Thus, my next plan is to improve the DDLSuite and HiveDDLSuite by creating an abstract class.
|
Test build #71401 has finished for PR 16587 at commit
|
|
The following test cases failed:
All these test cases are creating hive serde tables. We might need to create another SQLQueryTestSuite in the hive package. |
|
cc @cloud-fan @yhuai @hvanhovell Any comment? |
|
Test build #71407 has finished for PR 16587 at commit
|
| withTable("my_ext_tab") { | ||
| (("a", "b") :: Nil).toDF().write.parquet(tempDir.getCanonicalPath) | ||
| (1 to 10).map { i => (i, i) }.toDF("a", "b").createTempView("my_temp_tab") | ||
| sql(s"CREATE TABLE my_ext_tab using parquet LOCATION '$tempDir'") |
There was a problem hiding this comment.
Ah, this is why you asked me #16586 (comment). I just ran a test on Windows to help.
- truncate table - external table, temporary table, view (not allowed) *** FAILED *** (188 milliseconds)
org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark arget mpspark-9e70280d-56dc-4063-8f40-8e62fec18394;
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
Maybe, it'd be okay to just use toURI if this test is not supposed to test path variants.
| val db = tableDefinition.identifier.database.get | ||
| requireDbExists(db) | ||
| val table = tableDefinition.identifier.table | ||
| if (tableDefinition.provider.isDefined && tableDefinition.provider.get.toLowerCase == "hive") { |
There was a problem hiding this comment.
shall we put this in HiveOnlyCheck?
| .add("HelLo", "int", nullable = false) | ||
| .add("WoRLd", "int", nullable = true), | ||
| provider = Some("hive"), | ||
| provider = Some("parquet"), |
There was a problem hiding this comment.
shall we also use defaultProvider here?
|
Test build #71485 has started for PR 16587 at commit |
|
retest this please |
|
Test build #71495 has finished for PR 16587 at commit
|
|
Test build #71528 has finished for PR 16587 at commit
|
|
retest this please |
|
Test build #71797 has started for PR 16587 at commit |
|
retest this please |
|
Test build #3545 has finished for PR 16587 at commit
|
|
LGTM |
|
Thanks! Merging to master. |
…port is Not Enabled ### What changes were proposed in this pull request? It is weird to create Hive source tables when using InMemoryCatalog. We are unable to operate it. This PR is to block users to create Hive source tables. ### How was this patch tested? Fixed the test cases Author: gatorsmile <[email protected]> Closes apache#16587 from gatorsmile/blockHiveTable.
…port is Not Enabled ### What changes were proposed in this pull request? It is weird to create Hive source tables when using InMemoryCatalog. We are unable to operate it. This PR is to block users to create Hive source tables. ### How was this patch tested? Fixed the test cases Author: gatorsmile <[email protected]> Closes apache#16587 from gatorsmile/blockHiveTable.
4 participants
What changes were proposed in this pull request?
It is weird to create Hive source tables when using InMemoryCatalog. We are unable to operate it. This PR is to block users to create Hive source tables.
How was this patch tested?
Fixed the test cases