[SPARK-9738][SQL] remove FromUnsafe and add its codegen version to GenerateSafe #8029
Conversation
|
Test build #40165 has finished for PR 8029 at commit
|
|
retest this please. |
|
Test build #40166 has finished for PR 8029 at commit
|
There was a problem hiding this comment.
This could be used in generated MutableProjection, so we need the copy() here, or we should make sure that UnsafeRow will not be used with MutableProjection (the fallback version of new aggregation)
There was a problem hiding this comment.
I moved the copy here, just want to tighten up the scope that we only need to do copy when convert a unsafe row to safe row(according to the fact that we only copy nested struct in GenerateSafe not here before).
There was a problem hiding this comment.
SafeProjection may be not the only one projection that turn an UnsafeRow into InternalRow. For some operators that accept both safe row and unsafe row, that will use generated mutable projection, it will got crupted UTF8String if we remove the copy here.
There was a problem hiding this comment.
makes sense, thanks for your explanation!
|
LGTM, except the UTF8String change, we could leave that our of this PR and investigate more. |
|
Could you fill the description a few words? |
|
Test build #40234 has finished for PR 8029 at commit
|
|
LGTM, merging this into master and 1.5, thanks! |
…enerateSafe In #7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes #8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe (cherry picked from commit 106c078) Signed-off-by: Davies Liu <[email protected]>
…enerateSafe In apache#7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes apache#8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe
3 participants
In #7752 we added
FromUnsafeto convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already haveGenerateSafeto do the conversion which is codegened. So we should removeFromUnsafeand implement its codegen version inGenerateSafe.