-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-21338][SQL]implement isCascadingTruncateTable() method in AggregatedDialect #19256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
ok to test |
| } | ||
|
|
||
| override def isCascadingTruncateTable(): Option[Boolean] = { | ||
| dialects.flatMap(_.isCascadingTruncateTable).headOption |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why using the first one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both getCatalystType and getJDBCType(dt: DataType) use the first one. Also, in the class header, it has the following:
/**
- AggregatedDialect can unify multiple dialects into one virtual Dialect.
- Dialects are tried in order, and the first dialect that does not return a
- neutral element will will.
- @param dialects List of dialects.
*/
It has a typo, I guess it means "the first dialect that does not return a
neutral element will return"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is different from getCatalystType and getJDBCType . Instead, we should follow canHandle.
|
Test build #81848 has finished for PR 19256 at commit
|
|
retest this please |
|
BTW, could you update the PR title? |
|
Test build #81859 has finished for PR 19256 at commit
|
|
Thanks @gatorsmile |
|
Let me correct what I said above. The logics should be
|
|
Thanks @gatorsmile |
|
It looks good, but the actual code should be very simple if you are writing using the Scala way |
|
LGTM pending Jenkins |
|
Test build #81898 has finished for PR 19256 at commit
|
|
Thanks! Merged to master |
|
@gatorsmile Thanks a lot for your help!!! |
What changes were proposed in this pull request?
org.apache.spark.sql.jdbc.JdbcDialect's method:
def isCascadingTruncateTable(): Option[Boolean] = None
is not overriden in org.apache.spark.sql.jdbc.AggregatedDialect class.
Because of this issue, when you add more than one dialect Spark doesn't truncate table because isCascadingTruncateTable always returns default None for Aggregated Dialect.
Will implement isCascadingTruncateTable in AggregatedDialect class in this PR.
How was this patch tested?
In JDBCSuite, inside test("Aggregated dialects"), will add one line to test AggregatedDialect.isCascadingTruncateTable