Skip to content

Commit 25c2776

Browse files
jiangxb1987gatorsmile
authored andcommitted
[SPARK-23523][SQL][FOLLOWUP] Minor refactor of OptimizeMetadataOnlyQuery
## What changes were proposed in this pull request? Inside `OptimizeMetadataOnlyQuery.getPartitionAttrs`, avoid using `zip` to generate attribute map. Also include other minor update of comments and format. ## How was this patch tested? Existing test cases. Author: Xingbo Jiang <[email protected]> Closes #20693 from jiangxb1987/SPARK-23523.
1 parent 476a7f0 commit 25c2776

File tree

2 files changed

+4
-4
lines changed

2 files changed

+4
-4
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ case class OptimizeMetadataOnlyQuery(catalog: SessionCatalog) extends Rule[Logic
8383
private def getPartitionAttrs(
8484
partitionColumnNames: Seq[String],
8585
relation: LogicalPlan): Seq[Attribute] = {
86-
val attrMap = relation.output.map(_.name.toLowerCase(Locale.ROOT)).zip(relation.output).toMap
86+
val attrMap = relation.output.map(a => a.name.toLowerCase(Locale.ROOT) -> a).toMap
8787
partitionColumnNames.map { colName =>
8888
attrMap.getOrElse(colName.toLowerCase(Locale.ROOT),
8989
throw new AnalysisException(s"Unable to find the column `$colName` " +

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,9 +67,9 @@ case class HadoopFsRelation(
6767
}
6868
}
6969

70-
// When data schema and partition schema have the overlapped columns, the output
71-
// schema respects the order of data schema for the overlapped columns, but respect
72-
// the data types of partition schema
70+
// When data and partition schemas have overlapping columns, the output
71+
// schema respects the order of the data schema for the overlapping columns, and it
72+
// respects the data types of the partition schema.
7373
val schema: StructType = {
7474
StructType(dataSchema.map(f => overlappedPartCols.getOrElse(getColName(f), f)) ++
7575
partitionSchema.filterNot(f => overlappedPartCols.contains(getColName(f))))

0 commit comments

Comments
 (0)