-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster
Description
Is your feature request related to a problem or challenge?
When pushdown_filters is enabled, DF should be able to eliminate the subsequent filter.
When enabling the option for tpc-h benchmark, the FilterExec and l_shipdate projection is still present in the plans.
For example query 3 we can see the filter:
FilterExec: l_shipdate@3 > 9204
ParquetExec: file_groups={2 groups: [[...]]},
projection=[l_orderkey, l_extendedprice, l_discount, l_shipdate], predicate=l_shipdate@10 > 9204, pruning_predicate=l_shipdate_max@0 > 9204
Describe the solution you'd like
Remove the filter when.
We probably need to make some changes to TableProvider FileFormat to support removing the filter based on the file format.
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster