Skip to content

Eliminate filter when pushdown_filters is enabled #7688

@Dandandan

Description

@Dandandan

Is your feature request related to a problem or challenge?

When pushdown_filters is enabled, DF should be able to eliminate the subsequent filter.
When enabling the option for tpc-h benchmark, the FilterExec and l_shipdate projection is still present in the plans.

For example query 3 we can see the filter:

FilterExec: l_shipdate@3 > 9204
  ParquetExec: file_groups={2 groups: [[...]]},
    projection=[l_orderkey, l_extendedprice, l_discount, l_shipdate], predicate=l_shipdate@10 > 9204, pruning_predicate=l_shipdate_max@0 > 9204

Describe the solution you'd like

Remove the filter when.

We probably need to make some changes to TableProvider FileFormat to support removing the filter based on the file format.

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceMake DataFusion faster

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions