Skip to content

[MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 … #6085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 … #6085

wants to merge 2 commits into from

Conversation

@tellison
Copy link
Member

@tellison tellison commented May 12, 2015

…arch.

  • zSeries 64-bit Java reports its architecture as s390x, so enhance the 64-bit check to accommodate that value.
  • SizeEstimator can detect whether IBM Java is using compressed object pointers using info in the "java.vm.info" property, so will do a better job than failing on the HotSpot MBean and guessing.

@AmplabJenkins
Copy link

AmplabJenkins commented May 12, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 12, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 12, 2015

Test build #32510 has started for PR 6085 at commit 0968989.

@SparkQA
Copy link

SparkQA commented May 12, 2015

Test build #32510 has finished for PR 6085 at commit 0968989.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 12, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 12, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32510/
Test FAILed.

srowen
srowen reviewed May 12, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
Copy link
Member

@srowen srowen May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I merge your other change that introduces an enum for JVM vendor, can it be used here?

Copy link
Member Author

@tellison tellison May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to have the utility as there are a few more places coming where I'll need to know the JVM vendor.

Architecturally, are you ok with creating a public utility in launcher and having a dependency on it from core?

Feel free to commit these and I'll restructure on the fly, or I can take both back and submit multiple changes in a single pull request.

Copy link
Member

@srowen srowen May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Darn, right, we don't want to add that dependency. Well, I think for the moment these two changes are OK in that regard since adding a new dependency across these two isn't good.

Copy link
Member Author

@tellison tellison May 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you feel about me creating a getJavaVendor() utility in org.apache.spark.util.Utils rather than in-lining the checks?

It would, however, be a duplicate of the version proposed for the launcher which is not great.

Copy link
Member

@srowen srowen May 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the duplication of the snippet in launcher may be unavoidable for now. I think this PR can be left as is, and a refactoring can be added when it becomes necessary in another PR?

@srowen
Copy link
Member

srowen commented May 12, 2015

Test failure looks unrelated but it will test again on further changes anyway

@srowen
Copy link
Member

srowen commented May 13, 2015

Jenkins, retest this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 13, 2015

Test build #32614 has started for PR 6085 at commit 0968989.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32614/
Test FAILed.

@shaneknapp
Copy link
Contributor

shaneknapp commented May 13, 2015

i will retrigger this job once i restart jenkins

@shaneknapp
Copy link
Contributor

shaneknapp commented May 13, 2015

jenkins, test this please

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 13, 2015

Test build #32615 has started for PR 6085 at commit 0968989.

@SparkQA
Copy link

SparkQA commented May 13, 2015

Test build #32615 has finished for PR 6085 at commit 0968989.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32615/
Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 13, 2015

Test build #32625 has started for PR 6085 at commit 1b6ff6a.

@SparkQA
Copy link

SparkQA commented May 13, 2015

Test build #32625 has finished for PR 6085 at commit 1b6ff6a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 13, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32625/
Test PASSed.

@asfgit asfgit closed this in 3cd9ad2 May 13, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@tellison @jeanlyn
[MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 …
f63220c 
…arch.

 - zSeries 64-bit Java reports its architecture as s390x, so enhance the 64-bit check to accommodate that value.

 - SizeEstimator can detect whether IBM Java is using compressed object pointers using info in the "java.vm.info" property, so will do a better job than failing on the HotSpot MBean and guessing.

Author: Tim Ellison <[email protected]>

Closes apache#6085 from tellison/SizeEstimator and squashes the following commits:

1b6ff6a [Tim Ellison] Merge branch 'master' of https://github.com/apache/spark into SizeEstimator
0968989 [Tim Ellison] [MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 arch.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@tellison @jeanlyn
[MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 …
f99ad5d 
…arch.

 - zSeries 64-bit Java reports its architecture as s390x, so enhance the 64-bit check to accommodate that value.

 - SizeEstimator can detect whether IBM Java is using compressed object pointers using info in the "java.vm.info" property, so will do a better job than failing on the HotSpot MBean and guessing.

Author: Tim Ellison <[email protected]>

Closes apache#6085 from tellison/SizeEstimator and squashes the following commits:

1b6ff6a [Tim Ellison] Merge branch 'master' of https://github.com/apache/spark into SizeEstimator
0968989 [Tim Ellison] [MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 arch.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@tellison @nemccarthy
[MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 …
b5f0294 
…arch.

 - zSeries 64-bit Java reports its architecture as s390x, so enhance the 64-bit check to accommodate that value.

 - SizeEstimator can detect whether IBM Java is using compressed object pointers using info in the "java.vm.info" property, so will do a better job than failing on the HotSpot MBean and guessing.

Author: Tim Ellison <[email protected]>

Closes apache#6085 from tellison/SizeEstimator and squashes the following commits:

1b6ff6a [Tim Ellison] Merge branch 'master' of https://github.com/apache/spark into SizeEstimator
0968989 [Tim Ellison] [MINOR] Enhance SizeEstimator to detect IBM compressed refs and s390 arch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tellison @AmplabJenkins @SparkQA @srowen @shaneknapp