[SPARK-15811] [SQL] fix the Python UDF in Scala 2.10 #13717
Conversation
|
cc @JoshRosen |
|
@rxin Another thing I found is that the SerializationDebugger ran into a infinite loop (or very very slow) before the fix, I have to disable it to release that an Iterator can't be serialized. cc @JoshRosen for a jenkins build for 2.10 (we could merge it then verity that) |
|
Just to take a step back, Is the suite lacking coverage for this feature ?. This sort of thing should have been caught in the unit tests. |
|
@damnMeddlingKid This feature is covered well by unit tests (PR builder ran with Scala 2.11 and Hadoop 2.x), but we did not have a jenkins build to ran with Scala 2.0, will have one to run again master. |
| // Check to make sure that the UDF can be evaluated with only the input of this child. | ||
| udf.references.subsetOf(child.outputSet) | ||
| } | ||
| }.toArray // force the iterator to an array, that can't be serialized in Scala 2.10 |
There was a problem hiding this comment.
do u mean the iterator couldn't have been serialized in scala 2.10?
maybe say "Turn it into an array since iterators cannot be serialized in Scala 2.10"
|
Test build #60659 has finished for PR 13717 at commit
|
|
Test build #60685 has finished for PR 13717 at commit
|
|
Merging in master/2.0. |
## What changes were proposed in this pull request? Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that . ## How was this patch tested? Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build). Author: Davies Liu <[email protected]> Closes #13717 from davies/fix_udf_210. (cherry picked from commit ef43b4e) Signed-off-by: Reynold Xin <[email protected]>
4 participants
What changes were proposed in this pull request?
Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that .
How was this patch tested?
Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build).