Skip to content

[SPARK-11998] [SQL] [test-hadoop2.0] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark #9979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-11998] [SQL] [test-hadoop2.0] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark #9979

wants to merge 1 commit into from

Conversation

@yhuai
Copy link
Contributor

@yhuai yhuai commented Nov 25, 2015

If we need to download Hive/Hadoop artifacts, try to download a Hadoop that matches the Hadoop used by Spark. If the Hadoop artifact cannot be resolved (e.g. Hadoop version is a vendor specific version like 2.0.0-cdh4.1.1), we will use Hadoop 2.4.0 (we used to hard code this version as the hadoop that we will download from maven) and we will not share Hadoop classes.

I tested this match in my laptop with the following confs (these confs are used by our builds). All tests are good.

build/sbt -Phadoop-1 -Dhadoop.version=1.2.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Phadoop-1 -Dhadoop.version=2.0.0-mr1-cdh4.1.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.2 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive

yhuai
yhuai reviewed Nov 25, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
Copy link
Contributor Author

@yhuai yhuai Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marmbrus This should be a val, right?

@yhuai
Copy link
Contributor Author

yhuai commented Nov 25, 2015

cc @marmbrus @JoshRosen for review.

@yhuai
Copy link
Contributor Author

yhuai commented Nov 26, 2015

what is the pic?

@yhuai yhuai changed the title [SPARK-11998] [SQL] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark [SPARK-11998] [SQL] [test-hadoop2.2] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark Nov 26, 2015
@yhuai
Copy link
Contributor Author

yhuai commented Nov 26, 2015

test this please

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #46716 has finished for PR 9979 at commit b3317a9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #46717 has finished for PR 9979 at commit 83e92ac.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor Author

yhuai commented Nov 26, 2015

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46722/consoleFull is the hadoop 2.2 run.

@JoshRosen
Copy link
Contributor

JoshRosen commented Nov 26, 2015

LGTM pending tests, per our offline discussion; this seems fine given that this auto-downloading of Hive classes isn't the recommended approach for production deployments, so the risks here don't seem huge.

andrewor14
andrewor14 reviewed Nov 26, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala Outdated
Copy link
Contributor

@andrewor14 andrewor14 Nov 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need this. It's a partial function

Copy link
Contributor Author

@yhuai yhuai Nov 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #46722 has finished for PR 9979 at commit 83e92ac.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
If we need to download Hive/Hadoop artifacts, try to download a Hadoo…
1f4605e 
…p that matches the Hadoop used by Spark. If the Hadoop artifact cannot be resolved (e.g. Hadoop version is a vendor specific version like 2.0.0-cdh4.1.1), we will use Hadoop 2.4.0 (we used to hard code this version as the hadoop that we will download from maven) and we will not share Hadoop classes.
@yhuai yhuai changed the title [SPARK-11998] [SQL] [test-hadoop2.2] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark [SPARK-11998] [SQL] [test-hadoop2.0] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark Nov 26, 2015
@yhuai
Copy link
Contributor Author

yhuai commented Nov 26, 2015

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46778/consoleFull is for hadoop 2.0 test.

@SparkQA
Copy link

SparkQA commented Nov 26, 2015

Test build #46778 has finished for PR 9979 at commit 1f4605e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor Author

yhuai commented Nov 27, 2015

OK. I am merging this to master and branch 1.6. I will watch the builds and see if there is any new issues.

@yhuai
Copy link
Contributor Author

yhuai commented Nov 27, 2015

hmm... not sure why https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46778/consoleFull used hadoop 2.3 profile.

@yhuai
Copy link
Contributor Author

yhuai commented Nov 27, 2015

Looks like there is a typo at https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L167-L170

asfgit pushed a commit that referenced this pull request Nov 27, 2015
@yhuai
[SPARK-11998][SQL][TEST-HADOOP2.0] When downloading Hadoop artifacts …
64813b7 
…from maven, we need to try to download the version that is used by Spark

If we need to download Hive/Hadoop artifacts, try to download a Hadoop that matches the Hadoop used by Spark. If the Hadoop artifact cannot be resolved (e.g. Hadoop version is a vendor specific version like 2.0.0-cdh4.1.1), we will use Hadoop 2.4.0 (we used to hard code this version as the hadoop that we will download from maven) and we will not share Hadoop classes.

I tested this match in my laptop with the following confs (these confs are used by our builds). All tests are good.
```
build/sbt -Phadoop-1 -Dhadoop.version=1.2.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Phadoop-1 -Dhadoop.version=2.0.0-mr1-cdh4.1.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.2 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive
```

Author: Yin Huai <[email protected]>

Closes #9979 from yhuai/versionsSuite.

(cherry picked from commit ad76562)
Signed-off-by: Yin Huai <[email protected]>
@asfgit asfgit closed this in ad76562 Nov 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yhuai @SparkQA @JoshRosen @andrewor14