[SPARK-10304][SQL]: throw error when the table directory is invalid #8547
Conversation
|
Test build #41855 has finished for PR 8547 at commit
|
There was a problem hiding this comment.
No. We need to preserve files like Parquet summary files (_metadata and _common_metadata).
There was a problem hiding this comment.
Thanks for the comments. It seems there is a lot of corner cases to be covered from the test case. for example 1st is valid, but 2nd is not:
1st:
"hdfs://host:9000/path/_temporary",
"hdfs://host:9000/path/a=10/b=20",
"hdfs://host:9000/path/_temporary/path",
2nd:
"hdfs://host:9000/path/_temporary",
"hdfs://host:9000/path/a=10/b=20",
"hdfs://host:9000/path/path1",
Adding an PartitionValues.empty does not solve the problem. Will close this PR, and investigate other approaches.
|
Adding an PartitionValues.empty does not cover all problems. Will close this PR, and investigate other approaches. |
…the dir structure is invalid JIRA: https://issues.apache.org/jira/browse/SPARK-10304 This patch detects if the structure of partition directories is not valid. The test cases are from #8547. Thanks zhzhan. cc liancheng Author: Liang-Chi Hsieh <[email protected]> Closes #8840 from viirya/detect_invalid_part_dir.
3 participants
Throw error if the directory of a table is invalid, validated by either all files in the directory are partitioned, or none of them are partitioned.
The consistency of detailed partition spec is validated by existing parsePartitionColumn.