[SPARK-21769] [SQL] Add a table-specific option for always respecting schemas inferred/controlled by Spark SQL #19003
Conversation
|
Test build #80882 has finished for PR 19003 at commit
|
|
Test build #80899 has finished for PR 19003 at commit
|
| | INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' | ||
| | OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' | ||
| |LOCATION '$location' | ||
| |TBLPROPERTIES ('avro.schema.literal' = '$avroSchema') |
There was a problem hiding this comment.
For such an example that requires users setting TBLPROPERTIES, it sounds like we are unable to use the CREATE TABLE USING command. cc @cloud-fan
There was a problem hiding this comment.
There was an argument about whether we should add TBLPROPERTIES, and we decided to not add it. I'm totally fine to add it if it's necessary.
|
LGTM |
|
retest this please |
| import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap | ||
|
|
||
| /** | ||
| * Options for the Parquet data source. |
|
LGTM, thanks! Are these table properties documented somewhere? |
|
We might need a dedicated section for documenting all the table-specific conf options. |
|
Test build #80999 has finished for PR 19003 at commit
|
|
Merging to master, thanks! |
|
Test build #81001 has finished for PR 19003 at commit
|
4 participants
What changes were proposed in this pull request?
For Hive-serde tables, we always respect the schema stored in Hive metastore, because the schema could be altered by the other engines that share the same metastore. Thus, we always trust the metastore-controlled schema for Hive-serde tables when the schemas are different (without considering the nullability and cases). However, in some scenarios, Hive metastore also could INCORRECTLY overwrite the schemas when the serde and Hive metastore built-in serde are different.
The proposed solution is to introduce a table-specific option for such scenarios. For a specific table, users can make Spark always respect Spark-inferred/controlled schema instead of trusting metastore-controlled schema. By default, we trust Hive metastore-controlled schema.
How was this patch tested?
Added a cross-version test case