Rewrite expression in FilterExec instead of the data

[adriangb]

Is your feature request related to a problem or challenge?

pydantic#35 (comment)

Describe the solution you'd like

We can do the same thing we started doing in ParquetOpener:

datafusion/datafusion/datasource-parquet/src/opener.rs

Line 265 in b7c014a

PhysicalExprSimplifier::new(&physical_file_schema).simplify(expr)

Describe alternatives you've considered

No response

Additional context

No response

[geetanshjuneja]

take

[adriangb]

In hindsight this may not be that simple: random() doesn't produce a literal but rather an array, so casting that may not be any cheaper than the data.

[alamb]

What might be worth doing is writing some tests before we write some code

I think the hard bit about writing tests for this particular feature is that this optimization is already done at the logical level

So for example, filters like this that would benefit

DataFusion CLI v49.0.0
> create table filter_test(string_col varchar, float_col float) as values ('foo', 1.0), ('bar', 2.0);
0 row(s) fetched.
Elapsed 0.007 seconds.

-- note we are comparing a string column to an integer, but datafusion
-- converts the integer to string (does not cast the column)
> explain format indent select * from filter_test where string_col = 1;
+---------------+-------------------------------------------------------------+
| plan_type     | plan                                                        |
+---------------+-------------------------------------------------------------+
| logical_plan  | Filter: filter_test.string_col = Utf8View("1")              |
|               |   TableScan: filter_test projection=[string_col, float_col] |
| physical_plan | CoalesceBatchesExec: target_batch_size=8192                 |
|               |   FilterExec: string_col@0 = 1                              |
|               |     DataSourceExec: partitions=1, partition_sizes=[1]       |
|               |                                                             |
+---------------+-------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.002 seconds.

I think to trigger the issue we would have to somehow push down expressions that weren't already simplified 🤔

[adriangb]

Yep agreed that's what I realized after making the issue. I think random() is a somewhat special case. So maybe we should just close this?

[alamb]

I think closing it is a good idea unless we can come up with an example 🤔

[alamb]

🎉