Skip to content

Commit 3e8975b

Browse files
panbingkunsrowen
authored andcommitted
[MINOR][SQL] Remove ColumnIOUtil
### What changes were proposed in this pull request? This PR aim to remove redundant ColumnIOUtil. ### Why are the changes needed? Complete todo. <img width="464" alt="image" src="https://user-images.githubusercontent.com/15246973/184788406-abcf6794-e623-4e64-8dc7-2ff1b05dcbed.png"> reason: from parquet version 1.12.3, methods below are public > 1.ColumnIO.getDefinitionLevel: https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-column/src/main/java/org/apache/parquet/io/ColumnIO.java#L84-L86 > 2.ColumnIO.getRepetitionLevel: https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-column/src/main/java/org/apache/parquet/io/ColumnIO.java#L77-L79 > 3.ColumnIO.getFieldPath: https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-column/src/main/java/org/apache/parquet/io/ColumnIO.java#L50-L52 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closes #37529 from panbingkun/remove_ColumnIOUtil. Authored-by: panbingkun <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 95aebcb commit 3e8975b

File tree

2 files changed

+6
-47
lines changed

2 files changed

+6
-47
lines changed

sql/core/src/main/java/org/apache/parquet/io/ColumnIOUtil.java

Lines changed: 0 additions & 40 deletions
This file was deleted.

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetColumn.scala

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
package org.apache.spark.sql.execution.datasources.parquet
1919

2020
import org.apache.parquet.column.ColumnDescriptor
21-
import org.apache.parquet.io.ColumnIOUtil
2221
import org.apache.parquet.io.GroupColumnIO
2322
import org.apache.parquet.io.PrimitiveColumnIO
2423
import org.apache.parquet.schema.Type.Repetition
@@ -42,14 +41,14 @@ case class ParquetColumn(
4241

4342
object ParquetColumn {
4443
def apply(sparkType: DataType, io: PrimitiveColumnIO): ParquetColumn = {
45-
this(sparkType, Some(io.getColumnDescriptor), ColumnIOUtil.getRepetitionLevel(io),
46-
ColumnIOUtil.getDefinitionLevel(io), io.getType.isRepetition(Repetition.REQUIRED),
47-
ColumnIOUtil.getFieldPath(io), Seq.empty)
44+
this(sparkType, Some(io.getColumnDescriptor), io.getRepetitionLevel,
45+
io.getDefinitionLevel, io.getType.isRepetition(Repetition.REQUIRED),
46+
io.getFieldPath, Seq.empty)
4847
}
4948

5049
def apply(sparkType: DataType, io: GroupColumnIO, children: Seq[ParquetColumn]): ParquetColumn = {
51-
this(sparkType, None, ColumnIOUtil.getRepetitionLevel(io),
52-
ColumnIOUtil.getDefinitionLevel(io), io.getType.isRepetition(Repetition.REQUIRED),
53-
ColumnIOUtil.getFieldPath(io), children)
50+
this(sparkType, None, io.getRepetitionLevel,
51+
io.getDefinitionLevel, io.getType.isRepetition(Repetition.REQUIRED),
52+
io.getFieldPath, children)
5453
}
5554
}

0 commit comments

Comments
 (0)