Skip to content

Commit 3aed305

Browse files
jerryshaotdas
authored andcommitted
[SPARK-4999][Streaming] Change storeInBlockManager to false by default
Currently WAL-backed block is read out from HDFS and put into BlockManger with storage level MEMORY_ONLY_SER by default, since WAL-backed block is already materialized in HDFS with fault-tolerance, no need to put into BlockManger again by default. Author: jerryshao <[email protected]> Closes apache#3906 from jerryshao/SPARK-4999 and squashes the following commits: b95f95e [jerryshao] Change storeInBlockManager to false by default
1 parent aff49a3 commit 3aed305

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ abstract class ReceiverInputDStream[T: ClassTag](@transient ssc_ : StreamingCont
8686
}.toArray
8787
// Since storeInBlockManager = false, the storage level does not matter.
8888
new WriteAheadLogBackedBlockRDD[T](ssc.sparkContext,
89-
blockIds, logSegments, storeInBlockManager = true, StorageLevel.MEMORY_ONLY_SER)
89+
blockIds, logSegments, storeInBlockManager = false, StorageLevel.MEMORY_ONLY_SER)
9090
} else {
9191
new BlockRDD[T](ssc.sc, blockIds)
9292
}

0 commit comments

Comments
 (0)