Skip to content

Commit 5bd306c

Browse files
gengliangwanggatorsmile
authored andcommitted
[SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2
## What changes were proposed in this pull request? Revise doc of method pushFilters in SupportsPushDownFilters/SupportsPushDownCatalystFilters In `FileSourceStrategy`, except `partitionKeyFilters`(the references of which is subset of partition keys), all filters needs to be evaluated after scanning. Otherwise, Spark will get wrong result from data sources like Orc/Parquet. This PR is to improve the doc. Author: Wang Gengliang <[email protected]> Closes #20769 from gengliangwang/revise_pushdown_doc. (cherry picked from commit 10b0657) Signed-off-by: gatorsmile <[email protected]>
1 parent b083bd1 commit 5bd306c

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434
public interface SupportsPushDownCatalystFilters extends DataSourceReader {
3535

3636
/**
37-
* Pushes down filters, and returns unsupported filters.
37+
* Pushes down filters, and returns filters that need to be evaluated after scanning.
3838
*/
3939
Expression[] pushCatalystFilters(Expression[] filters);
4040

sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
public interface SupportsPushDownFilters extends DataSourceReader {
3333

3434
/**
35-
* Pushes down filters, and returns unsupported filters.
35+
* Pushes down filters, and returns filters that need to be evaluated after scanning.
3636
*/
3737
Filter[] pushFilters(Filter[] filters);
3838

0 commit comments

Comments
 (0)