Skip to content

[SPARK-16060][SQL] Support Vectorized ORC Reader #19943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 23 commits into from
Closed

[SPARK-16060][SQL] Support Vectorized ORC Reader #19943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0a0d7ad
[SPARK-16060][SQL] Support Vectorized ORC Reader
dongjoon-hyun Dec 6, 2017
42048cc
Address comments
dongjoon-hyun Dec 12, 2017
c0b9aea
Address more comments.
dongjoon-hyun Dec 12, 2017
a65ea04
Address comments.
dongjoon-hyun Dec 28, 2017
56087ac
Revert the change on DataSourceScanExec.
dongjoon-hyun Dec 28, 2017
e52fd27
Rebased to the master and address some comments.
dongjoon-hyun Jan 2, 2018
e9675e7
Update benchmarks and address comments.
dongjoon-hyun Jan 6, 2018
1e6ca5c
Address comments.
dongjoon-hyun Jan 6, 2018
803c1d1
fix
dongjoon-hyun Jan 6, 2018
7214ec0
Add JavaOrcColumnarBatchReader and benchmark.
dongjoon-hyun Jan 7, 2018
3a0702a
Use putX instead of appendX and address other comments.
dongjoon-hyun Jan 8, 2018
3df7d1e
Split nextBatch.
dongjoon-hyun Jan 8, 2018
15cac9c
Use `putArray`.
dongjoon-hyun Jan 8, 2018
e3d6342
Revert putArray bacause BytesColumnVector doesn't use a single base b…
dongjoon-hyun Jan 8, 2018
10e5d7a
Remove scala version
dongjoon-hyun Jan 8, 2018
ba03d20
Remove `putByteArray`.
dongjoon-hyun Jan 8, 2018
b623ca4
fix typo.
dongjoon-hyun Jan 9, 2018
8fc2162
Split functions.
dongjoon-hyun Jan 9, 2018
91b3d66
Address comments.
dongjoon-hyun Jan 9, 2018
db02555
Use recordReader.getProgress
dongjoon-hyun Jan 9, 2018
8abbbc3
Revert "Address comments."
dongjoon-hyun Jan 9, 2018
42182e3
Revert "Split functions."
dongjoon-hyun Jan 9, 2018
2cf98b6
Remove totalRowCount/rowsReturned and address comments..
dongjoon-hyun Jan 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,11 @@ object SQLConf {
.checkValues(Set("hive", "native"))
.createWithDefault("native")

val ORC_VECTORIZED_READER_ENABLED = buildConf("spark.sql.orc.enableVectorizedReader")
.doc("Enables vectorized orc decoding.")
.booleanConf
.createWithDefault(true)

val ORC_FILTER_PUSHDOWN_ENABLED = buildConf("spark.sql.orc.filterPushdown")
.doc("When true, enable filter pushdown for ORC files.")
.booleanConf
Expand Down Expand Up @@ -1183,6 +1188,8 @@ class SQLConf extends Serializable with Logging {

def orcCompressionCodec: String = getConf(ORC_COMPRESSION)

def orcVectorizedReaderEnabled: Boolean = getConf(ORC_VECTORIZED_READER_ENABLED)

def parquetCompressionCodec: String = getConf(PARQUET_COMPRESSION)

def parquetVectorizedReaderEnabled: Boolean = getConf(PARQUET_VECTORIZED_READER_ENABLED)
Expand Down
Loading