[SPARK-12841][SQL] fix cast in filter #10781
Conversation
|
retest this please |
|
Test build #49506 has finished for PR 10781 at commit
|
cloud-fan
changed the title
cloud-fan
changed the title
cloud-fan
changed the title
There was a problem hiding this comment.
We use transformUp at here to make this logic equivalent with the original logic that we have in the cast, right (since we want to fix it in 1.6)? (in future, will it be possible to just use cast Cast(ne: NamedExpression, to) => UnresolvedAlias(Cast(ne, to))?)
There was a problem hiding this comment.
hmm, how about something like df.select($"a".cast("int").cast("string"))? Do we need to propagate the name to the top level?
There was a problem hiding this comment.
oh, i see. It will be good to propagate the name to the top level. The current change can do the job, right?
There was a problem hiding this comment.
yea, it works, I'll add a test for it.
|
Test build #49604 has finished for PR 10781 at commit
|
|
Test build #49612 has finished for PR 10781 at commit
|
|
ping @yhuai |
|
LGTM. Merging to master and branch 1.6. |
|
there are a few conflicts with branch 1.6. Can you create a pr for the backport? Thanks! |
|
yea, sure |
In SPARK-10743 we wrap cast with `UnresolvedAlias` to give `Cast` a better alias if possible. However, for cases like filter, the `UnresolvedAlias` can't be resolved and actually we don't need a better alias for this case. This PR move the cast wrapping logic to `Column.named` so that we will only do it when we need a alias name. backport #10781 to 1.6 Author: Wenchen Fan <[email protected]> Closes #10819 from cloud-fan/bug.
3 participants
In SPARK-10743 we wrap cast with
UnresolvedAliasto giveCasta better alias if possible. However, for cases likefilter, theUnresolvedAliascan't be resolved and actually we don't need a better alias for this case. This PR move the cast wrapping logic toColumn.namedso that we will only do it when we need a alias name.