Skip to content

Commit 2769633

Browse files
committed
Address comment.
1 parent 46c8697 commit 2769633

File tree

4 files changed

+21
-7
lines changed

4 files changed

+21
-7
lines changed

docs/sql-programming-guide.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1006,7 +1006,12 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession
10061006

10071007
## ORC Files
10081008

1009-
Since Spark 2.3, Spark supports a vectorized ORC reader with a new ORC file format for ORC files. To do that, the following configurations are newly added. The vectorized reader is used for the native ORC tables (e.g., the ones created using the clause `USING ORC`) when `spark.sql.orc.impl` is set to `native` and `spark.sql.orc.enableVectorizedReader` is set to `true`. For the Hive ORC serde table (e.g., the ones created using the clause `USING HIVE OPTIONS (fileFormat 'ORC')`), the vectorized reader is used when `spark.sql.hive.convertMetastoreOrc` is set to `true`.
1009+
Since Spark 2.3, Spark supports a vectorized ORC reader with a new ORC file format for ORC files.
1010+
To do that, the following configurations are newly added. The vectorized reader is used for the
1011+
native ORC tables (e.g., the ones created using the clause `USING ORC`) when `spark.sql.orc.impl`
1012+
is set to `native` and `spark.sql.orc.enableVectorizedReader` is set to `true`. For the Hive ORC
1013+
serde tables (e.g., the ones created using the clause `USING HIVE OPTIONS (fileFormat 'ORC')`),
1014+
the vectorized reader is used when `spark.sql.hive.convertMetastoreOrc` is set to `true`.
10101015

10111016
<table class="table">
10121017
<tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th></tr>

sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,11 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext with Befo
3636
}
3737

3838
override def afterAll(): Unit = {
39-
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
40-
super.afterAll()
39+
try {
40+
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
41+
} finally {
42+
super.afterAll()
43+
}
4144
}
4245

4346
private val allFileBasedDataSources = Seq("orc", "parquet", "csv", "json", "text")

sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,11 @@ class FileStreamSinkSuite extends StreamTest {
3939
}
4040

4141
override def afterAll(): Unit = {
42-
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
43-
super.afterAll()
42+
try {
43+
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
44+
} finally {
45+
super.afterAll()
46+
}
4447
}
4548

4649
test("unpartitioned writing and batch reading") {

sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -213,8 +213,11 @@ class FileStreamSourceSuite extends FileStreamSourceTest {
213213
}
214214

215215
override def afterAll(): Unit = {
216-
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
217-
super.afterAll()
216+
try {
217+
spark.sessionState.conf.unsetConf(SQLConf.ORC_IMPLEMENTATION)
218+
} finally {
219+
super.afterAll()
220+
}
218221
}
219222

220223
// ============= Basic parameter exists tests ================

0 commit comments

Comments
 (0)