[SPARK-13922][SQL] Filter rows with null attributes in vectorized parquet reader #11749
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
cc @nongli |
|
Test build #53253 has finished for PR 11749 at commit
|
Contributor
There was a problem hiding this comment.
Instead of commenting it this way, can you put the fraction in the benchmark name?
e.g. String with Nulls Scan (95%)
Contributor
|
Can you add some test cases to columnarbatchsuite that exercises this? |
sameeragarwal
force-pushed
the
perf-testing
branch
from
af217fe to
2d1066f
Compare
Member
Author
|
thanks, all comments addressed! |
sameeragarwal
changed the title
|
Test build #53338 has finished for PR 11749 at commit
|
| * attribute is filtered out. | ||
| */ | ||
| public final void filterNullsInColumn(int ordinal) { | ||
| assert(!nullFilteredColumns.contains(ordinal)); |
Contributor
There was a problem hiding this comment.
I don't think this assert is necessary. I think this is perfectly valid and makes this api easier to use.
Member
Author
There was a problem hiding this comment.
yes, with the set it's not longer required. Fixed.
Contributor
|
LGTM |
|
Test build #53354 has finished for PR 11749 at commit
|
roygao94
pushed a commit
to roygao94/spark
that referenced
this pull request
Mar 22, 2016
…quet reader
# What changes were proposed in this pull request?
It's common for many SQL operators to not care about reading `null` values for correctness. Currently, this is achieved by performing `isNotNull` checks (for all relevant columns) on a per-row basis. Pushing these null filters in the vectorized parquet reader should bring considerable benefits (especially for cases when the underlying data doesn't contain any nulls or contains all nulls).
## How was this patch tested?
Intel(R) Core(TM) i7-4960HQ CPU 2.60GHz
String with Nulls Scan (0%): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
SQL Parquet Vectorized 1229 / 1648 8.5 117.2 1.0X
PR Vectorized 833 / 846 12.6 79.4 1.5X
PR Vectorized (Null Filtering) 732 / 782 14.3 69.8 1.7X
Intel(R) Core(TM) i7-4960HQ CPU 2.60GHz
String with Nulls Scan (50%): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
SQL Parquet Vectorized 995 / 1053 10.5 94.9 1.0X
PR Vectorized 732 / 772 14.3 69.8 1.4X
PR Vectorized (Null Filtering) 725 / 790 14.5 69.1 1.4X
Intel(R) Core(TM) i7-4960HQ CPU 2.60GHz
String with Nulls Scan (95%): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
SQL Parquet Vectorized 326 / 333 32.2 31.1 1.0X
PR Vectorized 190 / 200 55.1 18.2 1.7X
PR Vectorized (Null Filtering) 168 / 172 62.2 16.1 1.9X
Author: Sameer Agarwal <[email protected]>
Closes apache#11749 from sameeragarwal/perf-testing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
It's common for many SQL operators to not care about reading
nullvalues for correctness. Currently, this is achieved by performingisNotNullchecks (for all relevant columns) on a per-row basis. Pushing these null filters in the vectorized parquet reader should bring considerable benefits (especially for cases when the underlying data doesn't contain any nulls or contains all nulls).How was this patch tested?