SKIPME merged Apache branch-1.6 and a couple more upstream integration changes #125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author: Yadong Qi <[email protected]> Closes apache#10096 from watermen/patch-1. (cherry picked from commit d0d7ec5) Signed-off-by: Reynold Xin <[email protected]>
…laDoc This fixes SPARK-12000, verified on my local with JDK 7. It seems that `scaladoc` try to match method names and messed up with annotations. cc: JoshRosen jkbradley Author: Xiangrui Meng <[email protected]> Closes apache#10114 from mengxr/SPARK-12000.2. (cherry picked from commit 9bb695b) Signed-off-by: Xiangrui Meng <[email protected]>
…ritySuite We should try increasing a timeout in NettyBlockTransferSecuritySuite in order to reduce that suite's flakiness in Jenkins. Author: Josh Rosen <[email protected]> Closes apache#10113 from JoshRosen/SPARK-12082. (cherry picked from commit ae40253) Signed-off-by: Reynold Xin <[email protected]>
…toString. https://issues.apache.org/jira/browse/SPARK-12109 The change of https://issues.apache.org/jira/browse/SPARK-11596 exposed the problem. In the sql plan viz, the filter shows  After changes in this PR, the viz is back to normal.  Author: Yin Huai <[email protected]> Closes apache#10111 from yhuai/SPARK-12109. (cherry picked from commit ec2b6c2) Signed-off-by: Reynold Xin <[email protected]>
In Java Spec java.sql.Connection, it has boolean getAutoCommit() throws SQLException Throws: SQLException - if a database access error occurs or this method is called on a closed connection So if conn.getAutoCommit is called on a closed connection, a SQLException will be thrown. Even though the code catch the SQLException and program can continue, I think we should check conn.isClosed before calling conn.getAutoCommit to avoid the unnecessary SQLException. Author: Huaxin Gao <[email protected]> Closes apache#10095 from huaxingao/spark-12088. (cherry picked from commit 5349851) Signed-off-by: Sean Owen <[email protected]>
\cc mengxr Author: Jeff Zhang <[email protected]> Closes apache#10093 from zjffdu/mllib_typo. (cherry picked from commit 7470d9e) Signed-off-by: Sean Owen <[email protected]>
this is to fix some typo in external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala Author: microwishing <[email protected]> Closes apache#10121 from microwishing/master. (cherry picked from commit 95b3cf1) Signed-off-by: Sean Owen <[email protected]>
…conflicts with dplyr shivaram Author: felixcheung <[email protected]> Closes apache#10119 from felixcheung/rdocdplyrmasked. (cherry picked from commit 43c575c) Signed-off-by: Shivaram Venkataraman <[email protected]>
…ition met in Master Downgrade to warning log for unexpected state transition. andrewor14 please review, thanks a lot. Author: jerryshao <[email protected]> Closes apache#10091 from jerryshao/SPARK-12059. (cherry picked from commit 7bc9e1d) Signed-off-by: Andrew Or <[email protected]>
…r and AppClient `SynchronousQueue` cannot cache any task. This issue is similar to apache#9978. It's an easy fix. Just use the fixed `ThreadUtils.newDaemonCachedThreadPool`. Author: Shixiong Zhu <[email protected]> Closes apache#10108 from zsxwing/fix-threadpool. (cherry picked from commit 649be4f) Signed-off-by: Shixiong Zhu <[email protected]>
**Problem.** Event logs in 1.6 were much bigger than 1.5. I ran page rank and the event log size in 1.6 was almost 5x that in 1.5. I did a bisect to find that the RDD callsite added in apache#9398 is largely responsible for this. **Solution.** This patch removes the long form of the callsite (which is not used!) from the event log. This reduces the size of the event log significantly. *Note on compatibility*: if this patch is to be merged into 1.6.0, then it won't break any compatibility. Otherwise, if it is merged into 1.6.1, then we might need to add more backward compatibility handling logic (currently does not exist yet). Author: Andrew Or <[email protected]> Closes apache#10115 from andrewor14/smaller-event-logs. (cherry picked from commit 688e521) Signed-off-by: Andrew Or <[email protected]>
Use ```coefficients``` replace ```weights```, I wish they are the last two. mengxr Author: Yanbo Liang <[email protected]> Closes apache#10065 from yanboliang/coefficients. (cherry picked from commit d576e76) Signed-off-by: Xiangrui Meng <[email protected]>
I haven't created a JIRA. If we absolutely need one I'll do it, but I'm fine with not getting mentioned in the release notes if that's the only purpose it'll serve. cc marmbrus - We should include this in 1.6-RC2 if there is one. I can open a second PR against branch-1.6 if necessary. Author: Nicholas Chammas <[email protected]> Closes apache#10109 from nchammas/spark-ec2-versions. (cherry picked from commit ad7cea6) Signed-off-by: Shivaram Venkataraman <[email protected]>
…tdown after test Author: Tathagata Das <[email protected]> Closes apache#10124 from tdas/InputStreamSuite-flaky-test. (cherry picked from commit a02d472) Signed-off-by: Tathagata Das <[email protected]>
…ck param and fix doc and add tests. Spark submit expects comma-separated list Author: felixcheung <[email protected]> Closes apache#10034 from felixcheung/sparkrinitdoc. (cherry picked from commit 2213441) Signed-off-by: Shivaram Venkataraman <[email protected]>
…tConf. TaskAttemptContext's constructor will clone the configuration instead of referencing it. Calling setConf after creating TaskAttemptContext makes any changes to the configuration made inside setConf unperceived by RecordReader instances. As an example, Titan's InputFormat will change conf when calling setConf. They wrap their InputFormat around Cassandra's ColumnFamilyInputFormat, and append Cassandra's configuration. This change fixes the following error when using Titan's CassandraInputFormat with Spark: *java.lang.RuntimeException: org.apache.thrift.protocol.TProtocolException: Required field 'keyspace' was not present! Struct: set_key space_args(keyspace:null)* There's a discussion of this error here: https://groups.google.com/forum/#!topic/aureliusgraphs/4zpwyrYbGAE Author: Anderson de Andrade <[email protected]> Closes apache#10046 from adeandrade/newhadooprdd-fix. (cherry picked from commit f434f36) Signed-off-by: Marcelo Vanzin <[email protected]>
… same name. Author: Sun Rui <[email protected]> Closes apache#10118 from sun-rui/SPARK-12104. (cherry picked from commit 5011f26) Signed-off-by: Shivaram Venkataraman <[email protected]>
…fter recovering StreamingContext from checkpoint Author: Tathagata Das <[email protected]> Closes apache#10127 from tdas/SPARK-12122. (cherry picked from commit 4106d80) Signed-off-by: Tathagata Das <[email protected]>
…ferenced When the spillable sort iterator was spilled, it was mistakenly keeping the last page in memory rather than the current page. This causes the current record to get corrupted. Author: Nong <[email protected]> Closes apache#10142 from nongli/spark-12089. (cherry picked from commit 95296d9) Signed-off-by: Davies Liu <[email protected]>
Python tests require access to the `KinesisTestUtils` file. When this file exists under src/test, python can't access it, since it is not available in the assembly jar. However, if we move KinesisTestUtils to src/main, we need to add the KinesisProducerLibrary as a dependency. In order to avoid this, I moved KinesisTestUtils to src/main, and extended it with ExtendedKinesisTestUtils which is under src/test that adds support for the KPL. cc zsxwing tdas Author: Burak Yavuz <[email protected]> Closes apache#10050 from brkyvz/kinesis-py.
…s in SparkR. Author: Sun Rui <[email protected]> Closes apache#9804 from sun-rui/SPARK-11774. (cherry picked from commit c8d0e16) Signed-off-by: Shivaram Venkataraman <[email protected]>
Need to match existing method signature Author: felixcheung <[email protected]> Closes apache#9680 from felixcheung/rcorr. (cherry picked from commit 895b6c4) Signed-off-by: Shivaram Venkataraman <[email protected]>
… be consistent with Scala/Python Change ```numPartitions()``` to ```getNumPartitions()``` to be consistent with Scala/Python. <del>Note: If we can not catch up with 1.6 release, it will be breaking change for 1.7 that we also need to explain in release note.<del> cc sun-rui felixcheung shivaram Author: Yanbo Liang <[email protected]> Closes apache#10123 from yanboliang/spark-12115. (cherry picked from commit 6979edf) Signed-off-by: Shivaram Venkataraman <[email protected]>
1, Add ```isNaN``` to ```Column``` for SparkR. ```Column``` should has three related variable functions: ```isNaN, isNull, isNotNull```. 2, Replace ```DataFrame.isNaN``` with ```DataFrame.isnan``` at SparkR side. Because ```DataFrame.isNaN``` has been deprecated and will be removed at Spark 2.0. <del>3, Add ```isnull``` to ```DataFrame``` for SparkR. ```DataFrame``` should has two related functions: ```isnan, isnull```.<del> cc shivaram sun-rui felixcheung Author: Yanbo Liang <[email protected]> Closes apache#10037 from yanboliang/spark-12044. (cherry picked from commit b6e8e63) Signed-off-by: Shivaram Venkataraman <[email protected]>
Author: gcc <[email protected]> Closes apache#10101 from rh99/master. (cherry picked from commit 04b6799) Signed-off-by: Sean Owen <[email protected]>
When \u appears in a comment block (i.e. in /**/), code gen will break. So, in Expression and CodegenFallback, we escape \u to \\u. yhuai Please review it. I did reproduce it and it works after the fix. Thanks! Author: gatorsmile <[email protected]> Closes apache#10155 from gatorsmile/escapeU. (cherry picked from commit 49efd03) Signed-off-by: Yin Huai <[email protected]>
markhamstra
added a commit
that referenced
this pull request
Dec 7, 2015
SKIPME merged Apache branch-1.6 and a couple more upstream integration changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.