Skip to content

[SPARK-14298] [ML] [MLlib] LDA should support disable checkpoint #12089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-14298] [ML] [MLlib] LDA should support disable checkpoint #12089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a5c0343
LDA should support disable checkpoint
yanboliang Mar 31, 2016
4af96d8
Disable checkpoint in PeriodicCheckpointer
yanboliang Apr 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicCheckpointer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ import org.apache.spark.storage.StorageLevel
* - This class removes checkpoint files once later Datasets have been checkpointed.
* However, references to the older Datasets will still return isCheckpointed = true.
*
* @param checkpointInterval Datasets will be checkpointed at this interval
* @param checkpointInterval Datasets will be checkpointed at this interval.
* If this interval was set as -1, then checkpointing will be disabled.
* @param sc SparkContext for the Datasets given to this checkpointer
* @tparam T Dataset type, such as RDD[Double]
*/
Expand Down Expand Up @@ -89,7 +90,8 @@ private[mllib] abstract class PeriodicCheckpointer[T](
updateCount += 1

// Handle checkpointing (after persisting)
if ((updateCount % checkpointInterval) == 0 && sc.getCheckpointDir.nonEmpty) {
if (checkpointInterval != -1 && (updateCount % checkpointInterval) == 0
&& sc.getCheckpointDir.nonEmpty) {
// Add new checkpoint before removing old checkpoints.
checkpoint(newData)
checkpointQueue.enqueue(newData)
Expand Down
3 changes: 2 additions & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ import org.apache.spark.storage.StorageLevel
* // checkpointed: graph4
* }}}
*
* @param checkpointInterval Graphs will be checkpointed at this interval
* @param checkpointInterval Graphs will be checkpointed at this interval.
* If this interval was set as -1, then checkpointing will be disabled.
* @tparam VD Vertex descriptor type
* @tparam ED Edge descriptor type
*
Expand Down