[SPARK-17356][SQL][1.6] Fix out of memory issue when generating JSON for TreeNode #14973
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clockfly
force-pushed
the
json_oom_1.6
branch
from
0b22ec6 to
b0b4b9e
Compare
Contributor
|
Dumb question, but aren't we using this in MLLib? |
Contributor
Author
|
@hvanhovell, the meta data is still kept in the plan. MLLib doesn't use toJson directly. |
|
Test build #64992 has finished for PR 14973 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Sep 6, 2016
…for TreeNode This is a backport of PR #14915 to branch 1.6. ## What changes were proposed in this pull request? class `org.apache.spark.sql.types.Metadata` is widely used in mllib to store some ml attributes. `Metadata` is commonly stored in `Alias` expression. ``` case class Alias(child: Expression, name: String)( val exprId: ExprId = NamedExpression.newExprId, val qualifier: Option[String] = None, val explicitMetadata: Option[Metadata] = None, override val isGenerated: java.lang.Boolean = false) ``` The `Metadata` can take a big memory footprint since the number of attributes is big ( in scale of million). When `toJSON` is called on `Alias` expression, the `Metadata` will also be converted to a big JSON string. If a plan contains many such kind of `Alias` expressions, it may trigger out of memory error when `toJSON` is called, since converting all `Metadata` references to JSON will take huge memory. With this PR, we will skip scanning Metadata when doing JSON conversion. For a reproducer of the OOM, and analysis, please look at jira https://issues.apache.org/jira/browse/SPARK-17356. ## How was this patch tested? Existing tests. Author: Sean Zhong <[email protected]> Closes #14973 from clockfly/json_oom_1.6.
Contributor
|
LGTM, merging to 1.6! |
Contributor
|
Thanks! |
zzcclp
pushed a commit
to zzcclp/spark
that referenced
this pull request
Sep 7, 2016
…for TreeNode This is a backport of PR apache#14915 to branch 1.6. ## What changes were proposed in this pull request? class `org.apache.spark.sql.types.Metadata` is widely used in mllib to store some ml attributes. `Metadata` is commonly stored in `Alias` expression. ``` case class Alias(child: Expression, name: String)( val exprId: ExprId = NamedExpression.newExprId, val qualifier: Option[String] = None, val explicitMetadata: Option[Metadata] = None, override val isGenerated: java.lang.Boolean = false) ``` The `Metadata` can take a big memory footprint since the number of attributes is big ( in scale of million). When `toJSON` is called on `Alias` expression, the `Metadata` will also be converted to a big JSON string. If a plan contains many such kind of `Alias` expressions, it may trigger out of memory error when `toJSON` is called, since converting all `Metadata` references to JSON will take huge memory. With this PR, we will skip scanning Metadata when doing JSON conversion. For a reproducer of the OOM, and analysis, please look at jira https://issues.apache.org/jira/browse/SPARK-17356. ## How was this patch tested? Existing tests. Author: Sean Zhong <[email protected]> Closes apache#14973 from clockfly/json_oom_1.6. (cherry picked from commit e6480a6)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a backport of PR #14915 to branch 1.6.
What changes were proposed in this pull request?
class
org.apache.spark.sql.types.Metadatais widely used in mllib to store some ml attributes.Metadatais commonly stored inAliasexpression.The
Metadatacan take a big memory footprint since the number of attributes is big ( in scale of million). WhentoJSONis called onAliasexpression, theMetadatawill also be converted to a big JSON string.If a plan contains many such kind of
Aliasexpressions, it may trigger out of memory error whentoJSONis called, since converting allMetadatareferences to JSON will take huge memory.With this PR, we will skip scanning Metadata when doing JSON conversion. For a reproducer of the OOM, and analysis, please look at jira https://issues.apache.org/jira/browse/SPARK-17356.
How was this patch tested?
Existing tests.