-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-31751][SQL]Serde property path overwrites hive table property location
#28882
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
TJX2014
wants to merge
7
commits into
apache:master
from
TJX2014:master-SPARK-31751-hive-table-location
Closed
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
87ff39b
spark.sql.follow.hive.table.location to compatible with legacy `path`…
TJX2014 b45d2b3
UT for hive table shared location
TJX2014 e51fd48
param loc from catalog module to hive module
TJX2014 2d5ba45
param to FOLLOW_TABLE_LOCATION
TJX2014 fc9143d
throw exception when location is not consistent
TJX2014 e44c1b9
toString => CatalogUtils.URIToString(_)
TJX2014 1496c66
compact SPARK-31061 test case and make getLocationFromStorageProps co…
TJX2014 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ import java.net.URI | |
| import org.apache.hadoop.conf.Configuration | ||
|
|
||
| import org.apache.spark.SparkConf | ||
| import org.apache.spark.sql.AnalysisException | ||
| import org.apache.spark.sql.catalyst.TableIdentifier | ||
| import org.apache.spark.sql.catalyst.catalog._ | ||
| import org.apache.spark.sql.execution.QueryExecutionException | ||
|
|
@@ -202,4 +203,26 @@ class HiveExternalCatalogSuite extends ExternalCatalogSuite { | |
| assert(alteredTable.provider === Some("foo")) | ||
| }) | ||
| } | ||
|
|
||
| test("SPARK-31751: serde property `path` overwrites hive table property location") { | ||
TJX2014 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| val catalog = newBasicCatalog() | ||
| val hiveTable = CatalogTable( | ||
| identifier = TableIdentifier("parq_alter", Some("db1")), | ||
| tableType = CatalogTableType.MANAGED, | ||
| storage = storageFormat, | ||
| schema = new StructType().add("col1", "int"), | ||
| provider = Some("parquet")) | ||
TJX2014 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| catalog.createTable(hiveTable, ignoreIfExists = false) | ||
| val beforeAlterTable = externalCatalog.getTable("db1", "parq_alter") | ||
| assert(beforeAlterTable.storage.locationUri.toString.contains("parq_alter")) | ||
|
|
||
| externalCatalog.client.runSqlHive( | ||
| "alter table db1.parq_alter rename to db1.parq_alter2") | ||
|
|
||
| val e = intercept[AnalysisException]( | ||
| externalCatalog.getTable("db1", "parq_alter2") | ||
| ) | ||
| assert(e.getMessage.contains("not equal to table prop path") | ||
| && e.getMessage.contains("parq_alter2")) | ||
| } | ||
|
Comment on lines
+219
to
+227
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We will get an exception when the path property is not consistent with storage location. |
||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit worried about string comparison. Are you sure the path string is always equal to the URI string? Shall we do normalization before comparing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We normalization URI as
toString => CatalogUtils.URIToString(_)may be better ?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to be safe, can we compare the URI? we can convert path string to URI with
CatalogUtils.stringToURI.