[SPARK-28456][SQL] Add a public API Encoder.makeCopy to allow creating Encoder without touching Scala Reflection
#25209
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zsxwing
changed the title
Encoder.copyEncoder to allow creating Encoder without touching Scala reflection.Encoder.copyEncoder to allow creating Encoder without touching Scala reflections
zsxwing
changed the title
Encoder.copyEncoder to allow creating Encoder without touching Scala reflectionsEncoder.copyEncoder to allow creating Encoder without touching Scala reflections
Member
Author
zsxwing
changed the title
Encoder.copyEncoder to allow creating Encoder without touching Scala reflectionsEncoder.copyEncoder to allow creating Encoder without touching Scala Reflection
|
Test build #107930 has finished for PR 25209 at commit
|
HyukjinKwon
reviewed
Jul 20, 2019
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
Outdated
Show resolved
Hide resolved
HyukjinKwon
reviewed
Jul 20, 2019
dongjoon-hyun
changed the title
Encoder.copyEncoder to allow creating Encoder without touching Scala ReflectionEncoder.copyEncoder to allow creating Encoder without touching Scala Reflection
zsxwing
changed the title
Encoder.copyEncoder to allow creating Encoder without touching Scala ReflectionEncoder.makeCopy to allow creating Encoder without touching Scala Reflection
|
Test build #107972 has finished for PR 25209 at commit
|
Contributor
|
thanks, merging to master! |
yiheng
pushed a commit
to yiheng/spark
that referenced
this pull request
Jul 24, 2019
…ing Encoder without touching Scala Reflection
## What changes were proposed in this pull request?
Because `Encoder` is not thread safe, the user cannot reuse an `Encoder` in multiple `Dataset`s. However, creating an `Encoder` for a complicated class is slow due to Scala Reflection. To eliminate the cost of Scala Reflection, right now I usually use the private API `ExpressionEncoder.copy` as follows:
```scala
object FooEncoder {
private lazy val _encoder: ExpressionEncoder[Foo] = ExpressionEncoder[Foo]()
implicit def encoder: ExpressionEncoder[Foo] = _encoder.copy()
}
```
This PR proposes a new method `makeCopy` in `Encoder` so that the above codes can be rewritten using public APIs.
```scala
object FooEncoder {
private lazy val _encoder: Encoder[Foo] = Encoders.product[Foo]()
implicit def encoder: Encoder[Foo] = _encoder.makeCopy
}
```
The method name is consistent with `TreeNode.makeCopy`.
## How was this patch tested?
Jenkins
Closes apache#25209 from zsxwing/encoder-copy.
Authored-by: Shixiong Zhu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Because
Encoderis not thread safe, the user cannot reuse anEncoderin multipleDatasets. However, creating anEncoderfor a complicated class is slow due to Scala Reflection. To eliminate the cost of Scala Reflection, right now I usually use the private APIExpressionEncoder.copyas follows:This PR proposes a new method
makeCopyinEncoderso that the above codes can be rewritten using public APIs.The method name is consistent with
TreeNode.makeCopy.How was this patch tested?
Jenkins