-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-17051][SQL] we should use hadoopConf in InsertIntoHiveTable #14634
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
cc @yhuai |
|
Test build #63746 has finished for PR 14634 at commit
|
|
Great! It resolves my original concern. Thanks! |
|
If it is controlled by hadoopConf, users might hit strange errors when using our DataFrameWriter APIs. |
I think it's hive only conf? Normal data source relation should not read this conf. |
|
If users want to use the Let me use a test case to show the issue. withTempDir { tmpDir =>
val basePath = tmpDir.getCanonicalPath
val externalTab = "extTable_with_partitions"
withTable(externalTab) {
assert(tmpDir.listFiles.isEmpty)
sql(
s"""
|CREATE EXTERNAL TABLE $externalTab (key INT, value STRING)
|PARTITIONED BY (ds STRING, hr STRING)
|stored as Parquet
|LOCATION '$basePath'
""".stripMargin)
// SQL interface
withSQLConf("hive.exec.dynamic.partition" -> "false") {
for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12")) {
sql(
s"""
|INSERT OVERWRITE TABLE $externalTab
|partition (ds='$ds',hr='$hr')
|SELECT 1, 'a'
""".stripMargin)
}
}
// DataFrameWriter interface
withSQLConf("hive.exec.dynamic.partition" -> "true",
"hive.exec.dynamic.partition.mode" -> "nonstrict") {
Seq((1, "2", "2008-04-09", "12")).toDF("key", "value", "ds", "hr").write
.insertInto(externalTab)
}
}
} |
|
Sorry, let me rephrase the potential issue.
|
|
|
uh, I see. Thank you! No more question. : ) |
|
Sorry. What's the necessity to make this change? |
|
Based on my understanding, after this PR, we will respect the conf values of |
|
@yhuai, @cloud-fan What's the status of this issue? Should this still be targeted for 2.0.1 (which it is currently in JIRA)? |
|
cc @yhuai , In |
|
retest this please |
|
@JoshRosen this is a regression in 2.0(it works in 1.6), so I think we should target it to 2.0 |
|
Test build #65620 has finished for PR 14634 at commit
|
|
This change looks good. Let's add a regression test. |
|
Test build #65629 has finished for PR 14634 at commit
|
|
LGTM. Merging to master and branch 2.0. |
## What changes were proposed in this pull request? Hive confs in hive-site.xml will be loaded in `hadoopConf`, so we should use `hadoopConf` in `InsertIntoHiveTable` instead of `SessionState.conf` ## How was this patch tested? N/A Author: Wenchen Fan <[email protected]> Closes #14634 from cloud-fan/bug. (cherry picked from commit eb004c6) Signed-off-by: Yin Huai <[email protected]>
What changes were proposed in this pull request?
Hive confs in hive-site.xml will be loaded in
hadoopConf, so we should usehadoopConfinInsertIntoHiveTableinstead ofSessionState.confHow was this patch tested?
N/A