[SPARK-17702][SQL] Code generation including too many mutable states exceeds JVM size limit. #15275
Conversation
|
Test build #66020 has finished for PR 15275 at commit
|
|
Test build #66025 has finished for PR 15275 at commit
|
|
Can you add more inline comments explaining what's going on? |
|
Test build #66090 has finished for PR 15275 at commit
|
|
LGTM. cc @davies to take a look too. |
| } | ||
|
|
||
| private def splitExpressions( | ||
| expressions: Seq[String], funcName: String, arguments: Seq[(String, String)]): String = { |
There was a problem hiding this comment.
How about exposing this API as public? This API looks more flexible and reusable.Is there any reason to declare as private?
(I replaced "non-public" with "public" in the first statement).
There was a problem hiding this comment.
As far as I know, there are two possible use cases of this API
https://issues.apache.org/jira/browse/SPARK-16223
https://github.com/apache/spark/pull/15219/files#diff-8bcc5aea39c73d4bf38aef6f6951d42cL587 in #15219
|
Test build #66267 has finished for PR 15275 at commit
|
|
Merging in master. Thanks. |
…exceeds JVM size limit. ## What changes were proposed in this pull request? Code generation including too many mutable states exceeds JVM size limit to extract values from `references` into fields in the constructor. We should split the generated extractions in the constructor into smaller functions. ## How was this patch tested? I added some tests to check if the generated codes for the expressions exceed or not. Author: Takuya UESHIN <[email protected]> Closes apache#15275 from ueshin/issues/SPARK-17702.
…exceeds JVM size limit. ## What changes were proposed in this pull request? Code generation including too many mutable states exceeds JVM size limit to extract values from `references` into fields in the constructor. We should split the generated extractions in the constructor into smaller functions. ## How was this patch tested? I added some tests to check if the generated codes for the expressions exceed or not. Author: Takuya UESHIN <[email protected]> Closes apache#15275 from ueshin/issues/SPARK-17702.
|
One important reason to prepare a backport for this is, as we deprecate the settings to turn off unsafe and codegen since 1.6, we can't turn them off to avoid JVM size limitation issue if users encounter that. |
|
It seems we can cherry-pick into |
|
Sorry to bother, it seems that the backport to 1.6 is not clean and easy. Any update on the decision of whether this fix will be tried to be backported to 1.6? thanks. |
|
When will this fix be backported to spark 2.0? |
7 participants
What changes were proposed in this pull request?
Code generation including too many mutable states exceeds JVM size limit to extract values from
referencesinto fields in the constructor.We should split the generated extractions in the constructor into smaller functions.
How was this patch tested?
I added some tests to check if the generated codes for the expressions exceed or not.