Skip to content

[SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1.6] Support AND/OR/IN/LIKE push-down filters for JDBC #16127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1.6] Support AND/OR/IN/LIKE push-down filters for JDBC #16127

wants to merge 1 commit into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Dec 3, 2016

This is rework from #10386 and add more tests and LIKE push-down support.

Author: Takeshi YAMAMURO [email protected]

Closes #10468 from maropu/SupportMorePushdownInJdbc.

(cherry picked from commit 5c2682b)

@maropu @dongjoon-hyun
[SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1.6] Support AN…
21fbe74 
…D/OR/IN/LIKE push-down filters for JDBC

This is rework from #10386 and add more tests and LIKE push-down support.

Author: Takeshi YAMAMURO <[email protected]>

Closes #10468 from maropu/SupportMorePushdownInJdbc.

(cherry picked from commit 5c2682b)
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

This is the second commit for #16021.
cc @gatorsmile .

dongjoon-hyun
dongjoon-hyun commented Dec 3, 2016
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
assert(sql("SELECT * FROM foobar").collect().size === 3)
}

test("SELECT * WHERE (simple predicates)") {
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Dec 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update the whole content of this testcase.

@rxin
Copy link
Contributor

rxin commented Dec 3, 2016

Why are we adding features? I thought they were bug fixes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

Ur, @rxin . We talked on #16021 about these 6 commits.
You delegated this review work to @gatorsmile , didn't you?

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

The solution of the bug fix was the feature of predicate pushdown to PostgreSQL in latest branch.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016 •
edited
Loading

My comment was

Hi, @rxin , @gatorsmile , @hvanhovell , @srowen .
Could you give some opinion? What I hoped here is having these 6 commits in branch-1.6.
But, if backports are not proper for this issue by the Apache Spark policy, I will just mark SPARK-18593 as Resolved with a fix version 2.0.0 and close this PR.
I know that It's not a good time for you to see this. Sorry for asking this.

@SparkQA
Copy link

SparkQA commented Dec 3, 2016

Test build #69604 has finished for PR 16127 at commit 21fbe74.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

Just let me know if this is not what you expect. I will close this and #16021 immediately as I mentioned before.
We just make a mistake one commit hopefully.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 3, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69604/
Test FAILed.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

Oh, my bad.

@rxin
Copy link
Contributor

rxin commented Dec 3, 2016

I didn't actually look at the content. I thought they were bug fixes and as a result said we should backport them. The title and description of the original pr implied they were bug fixes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

I see. If then, it's my fault.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Dec 3, 2016

I will close this now.

@dongjoon-hyun dongjoon-hyun mentioned this pull request Dec 3, 2016
Closed
@dongjoon-hyun dongjoon-hyun deleted the SPARK-12409-BACKPORT-1.6 branch April 27, 2018 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dongjoon-hyun @rxin @SparkQA @AmplabJenkins @maropu