Skip to content

[SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion #24068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 56 commits into from
Closed

[SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion #24068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
2527fee
WIP: optimize ORC predicate conversion
IvanVergiliev Jan 3, 2019
d6c408e
Implement isConvertibleToOrcPredicate with linear complexity
IvanVergiliev Feb 18, 2019
66f5064
Run FilterPushdownBenchmark with a smaller number of widths
IvanVergiliev Feb 19, 2019
1444c3c
fix formatting issues from merge
IvanVergiliev Feb 26, 2019
b662b60
Use Map.withDefault for the cache, and key it on (Filter, Boolean)
IvanVergiliev Feb 26, 2019
6336415
Use Map.getOrElseUpdate instead of WithDefault for actual caching; us…
IvanVergiliev Feb 26, 2019
e6b72a0
Override equals and hashCode to get to actual linear complexity
IvanVergiliev Feb 26, 2019
8a2c33f
Add a benchmark that tests only conversion and is able to expose line…
IvanVergiliev Feb 26, 2019
3f095e0
Add benchmarks for converting both from Column and from Expression
IvanVergiliev Mar 1, 2019
1f15026
Get rid of unnecessary objects in Expression benchmark
IvanVergiliev Mar 1, 2019
bf5e521
Don't use a raw SQL query in Column pushdown benchmark
IvanVergiliev Mar 1, 2019
15b3e5b
Add a benchmark case for Parquet Column pushdown
IvanVergiliev Mar 1, 2019
b0f8af2
Uncomment previously commented benchmarks
IvanVergiliev Mar 8, 2019
d37ffc0
Add comments everywhere
IvanVergiliev Mar 8, 2019
eb1b0c1
Only run benchmarks for a few pre-specified values
IvanVergiliev Mar 12, 2019
1e2651a
Add new benchmark results
IvanVergiliev Mar 12, 2019
78ad7b8
Key memoization table only based on
IvanVergiliev Mar 22, 2019
56d23cb
Implement tree trimming
IvanVergiliev May 7, 2019
265d088
Extract an inline method
IvanVergiliev May 7, 2019
a721e80
Add a TrimmedFilter value class for better type safety
IvanVergiliev May 8, 2019
fbe840a
Add Javadoc for the trimNonConvertibleFilters methods
IvanVergiliev May 8, 2019
5b649df
Apply changes to v2.3.4 subtree as well
IvanVergiliev May 8, 2019
9fe61d4
Add comments
IvanVergiliev May 8, 2019
c41421f
Replace if-else for And predicate with a pattern match
IvanVergiliev May 8, 2019
c3174e9
Minor formatting and style changes
IvanVergiliev May 8, 2019
9af9499
Code review comments
IvanVergiliev May 8, 2019
88d4181
Update Benchmark comments
IvanVergiliev May 9, 2019
5aa1502
Rename Expression -> Column in benchmark name
IvanVergiliev May 9, 2019
c6dacf8
Revert existing benchmark
IvanVergiliev May 15, 2019
d350bed
Initial implementation with filter-and-build in the same place
IvanVergiliev May 21, 2019
0f62282
Remove commented out code
IvanVergiliev May 21, 2019
90a781a
Get rid of the type member in ActionType
IvanVergiliev May 21, 2019
77af5b7
Remove unused imports
IvanVergiliev May 22, 2019
2b311d2
Add comments to new functions
IvanVergiliev May 27, 2019
1f2d5ec
Remove now unused TrimmedFilter class
IvanVergiliev May 27, 2019
ee25809
Apply changes to 2.3.4 version of the file
IvanVergiliev May 27, 2019
2befb28
Don't filter predicates in convertibleFilters since they are already …
IvanVergiliev May 27, 2019
9ea2a6f
Add OR partial pushdown comment back
IvanVergiliev May 27, 2019
b19792c
Don't explicitly set canPartialPushDownPredicates = true when filteri…
IvanVergiliev May 27, 2019
3d1b28e
Add back comment about wrapping leaf predicates in an AND
IvanVergiliev May 27, 2019
44d2583
Code review comments
IvanVergiliev May 28, 2019
9a6fd4d
Initial version that passes partial OR pushdown test
IvanVergiliev May 28, 2019
aa8a629
Make interface for filtering and building nicer
IvanVergiliev May 28, 2019
fa9bf70
Move actual conversion functionality into a separate class
IvanVergiliev May 28, 2019
caf013d
Improve filter interface
IvanVergiliev May 28, 2019
323b4e2
Fix comment
IvanVergiliev May 28, 2019
aeaf8a1
Add comment to updateBuilder
IvanVergiliev May 28, 2019
c12ed11
Code review comments
IvanVergiliev May 29, 2019
1735c0c
Code review comments
IvanVergiliev May 30, 2019
641b662
Style improvements to case-match statements
IvanVergiliev May 30, 2019
524a1e1
Remove performFilter method
IvanVergiliev May 31, 2019
1d1de5b
Add a doc for trimUnconvertibleFilters
IvanVergiliev Jun 5, 2019
3d5620d
Apply changes to hive/ subtree as well
IvanVergiliev Jun 6, 2019
67b7a86
Remove filter conversion benchmark
IvanVergiliev Jun 6, 2019
27ed4f9
Add a comment explaining what each large-filter benchmark does
IvanVergiliev Jun 6, 2019
24dcc24
Revert benchmark changes
IvanVergiliev Jun 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFiltersBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ trait OrcFiltersBase {

// Since ORC 1.5.0 (ORC-323), we need to quote for column names with `.` characters
// in order to distinguish predicate pushdown for nested columns.
protected def quoteAttributeNameIfNeeded(name: String) : String = {
protected[sql] def quoteAttributeNameIfNeeded(name: String) : String = {
if (!name.contains("`") && name.contains(".")) {
s"`$name`"
} else {
Expand All @@ -50,7 +50,7 @@ trait OrcFiltersBase {
* Return true if this is a searchable type in ORC.
* Both CharType and VarcharType are cleaned at AstBuilder.
*/
protected def isSearchableType(dataType: DataType) = dataType match {
protected[sql] def isSearchableType(dataType: DataType) = dataType match {
case BinaryType => false
case _: AtomicType => true
case _ => false
Expand Down
Loading