Skip to content

[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0 #15733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0 #15733

wants to merge 5 commits into from

Conversation

@srowen
Copy link
Member

@srowen srowen commented Nov 2, 2016

What changes were proposed in this pull request?

Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0. This does not actually implement any of the change in SPARK-18138, just peppers the documentation with notices about it.

How was this patch tested?

Doc build

@SparkQA
Copy link

SparkQA commented Nov 2, 2016

Test build #67980 has finished for PR 15733 at commit 61b4988.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

nchammas
nchammas reviewed Nov 2, 2016
View reviewed changes
Copy link
Contributor

@nchammas nchammas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java 7 and Python 2.6 support have both been deprecated since 2.0. Dunno if you wanna update the text to be more precise in this regard.

docs/building-spark.md Outdated

The Maven-based build is the build of reference for Apache Spark.
Building Spark using Maven requires Maven 3.3.9 or newer and Java 7+.
Note that support for Java 7 is deprecated as of Spark 2.1.0 and may be removed in Spark 2.2.0.
Copy link
Contributor

@nchammas nchammas Nov 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's been deprecated since 2.0.

Copy link
Member Author

@srowen srowen Nov 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will adjust this.

@SparkQA
Copy link

SparkQA commented Nov 2, 2016

Test build #67988 has finished for PR 15733 at commit 51fbb61.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

nchammas
nchammas reviewed Nov 2, 2016
View reviewed changes
docs/index.md Outdated
Note that support for Java 7, Python 2.6, Scala 2.10 and version of Hadoop before 2.6 are
deprecated as of Spark 2.1.0, and may be removed in Spark 2.2.0.
Note that support for Java 7 and Python 2.6 are deprecated as of Spark 2.0.0, and support for
Scala 2.10 and version of Hadoop before 2.6 are deprecated as of Spark 2.1.0, and may be
Copy link
Contributor

@nchammas nchammas Nov 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"... and versions of Hadoop..."

@SparkQA
Copy link

SparkQA commented Nov 2, 2016

Test build #67992 has finished for PR 15733 at commit 1ae3340.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Nov 2, 2016

Can you update SparkContext to print deprecation warning when these environments are used?

@srowen
Copy link
Member Author

srowen commented Nov 2, 2016

I added some warnings -- examples --

$ ./bin/spark-shell 
...
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
16/11/02 20:22:49 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
16/11/02 20:22:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...


$ ./bin/pyspark 
...
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
16/11/02 20:26:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/Users/srowen/Documents/spark/python/pyspark/context.py:192: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
  warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")
...

I found an API for Hadoop version info but it's labeled "private" and "unstable", so wasn't sure whether it's worth it to access it just to warn.

@SparkQA
Copy link

SparkQA commented Nov 2, 2016

Test build #68012 has finished for PR 15733 at commit 0d6092e.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

nchammas
nchammas reviewed Nov 2, 2016
View reviewed changes
Copy link
Contributor

@nchammas nchammas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a PySpark user is using Python 2.7+ and Java 7, will they see the warning about Java 7?

I think the answer is yes, but I just want to make sure.

@rxin
Copy link
Contributor

rxin commented Nov 2, 2016

LGTM pending tests.

@SparkQA
Copy link

SparkQA commented Nov 2, 2016

Test build #68013 has finished for PR 15733 at commit 8ee3d80.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen srowen mentioned this pull request Nov 3, 2016
Closed
@SparkQA
Copy link

SparkQA commented Nov 3, 2016

Test build #3408 has finished for PR 15733 at commit 8ee3d80.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Nov 4, 2016

Merging in master/branch-2.1.

asfgit pushed a commit that referenced this pull request Nov 4, 2016
@srowen @rxin
[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Had…
37550c4 
…oop < 2.6 are deprecated in Spark 2.1.0

## What changes were proposed in this pull request?

Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0. This does not actually implement any of the change in SPARK-18138, just peppers the documentation with notices about it.

## How was this patch tested?

Doc build

Author: Sean Owen <[email protected]>

Closes #15733 from srowen/SPARK-18138.

(cherry picked from commit dc4c600)
Signed-off-by: Reynold Xin <[email protected]>
@asfgit asfgit closed this in dc4c600 Nov 4, 2016
@srowen srowen deleted the SPARK-18138 branch November 5, 2016 10:00
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@srowen @uzadude
[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Had…
3912f9c 
…oop < 2.6 are deprecated in Spark 2.1.0

## What changes were proposed in this pull request?

Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0. This does not actually implement any of the change in SPARK-18138, just peppers the documentation with notices about it.

## How was this patch tested?

Doc build

Author: Sean Owen <[email protected]>

Closes apache#15733 from srowen/SPARK-18138.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@nchammas nchammas nchammas approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@srowen @SparkQA @rxin @nchammas