[SPARK-12409][SPARK-12387][SPARK-12391][SQL] Support AND/OR/IN/LIKE push-down filters for JDBC #10468
Conversation
There was a problem hiding this comment.
for clarity, it might be better to wrap it in parentheses
There was a problem hiding this comment.
Sorry, I'm not exactly sure what you point out.
You mean case Not(f) => s"(NOT (${compileFilter(f)}))"?
|
Test build #48297 has finished for PR 10468 at commit
|
There was a problem hiding this comment.
Should we compare using === instead of == here and elsewhere?
There was a problem hiding this comment.
I have no strong opinion on this.... is it better to fix them? ISTM collection types, e.g., set, need === comparisons.
There was a problem hiding this comment.
Does the newer version of scalatest library we use already uses macro so == and === are the same? Can you confirm? Anyway it's not that big of a deal here.
There was a problem hiding this comment.
Ah yes, now Assertions trait provides assert macro and the trait mixes in TripleEquals trait so we don't need to change == to ===.
|
LGTM - maybe we can merge this one once tests pass and then have @viirya rebase his new PR based on this? |
|
Test build #48447 has finished for PR 10468 at commit
|
|
Looks like you also need to update the test case. |
|
@rxin Good for me. |
|
Okay and I'll fix it in a minute. |
|
@rxin okay and we'll wait for tests done. |
|
Test build #48473 has finished for PR 10468 at commit
|
|
I've merged this. Thanks. |
… for JDBCRDD and add few filters This patch refactors the filter pushdown for JDBCRDD and also adds few filters. Added filters are basically from #10468 with some refactoring. Test cases are from #10468. Author: Liang-Chi Hsieh <[email protected]> Closes #10470 from viirya/refactor-jdbc-filter.
…IKE push-down filters for JDBC apache#10468
|
|
||
| // This is a test to reflect discussion in SPARK-12218. | ||
| // The older versions of spark have this kind of bugs in parquet data source. | ||
| val df1 = sql("SELECT * FROM foobar WHERE NOT (THEID != 2 AND NAME != 'mary')") |
There was a problem hiding this comment.
The two sub-conditions are both ok to be pushed down. So this doesn't actually test against the nested AND issue in SPARK-12218. See #19776
Btw, the two sub-conditions are filtered out the same rows. This doesn't reflect the issue too.
There was a problem hiding this comment.
yea, I think ok to drop this test.
5 participants
This is rework from #10386 and add more tests and LIKE push-down support.