Skip to content

Commit 0dc79ff

Browse files
Anthony Truchetsrowen
authored andcommitted
[SPARK-16440][MLLIB] Destroy broadcasted variables even on driver
## What changes were proposed in this pull request? Forgotten broadcasted variables were persisted into a previous #PR 14153). This PR turns those `unpersist()` into `destroy()` so that memory is freed even on the driver. ## How was this patch tested? Unit Tests in Word2VecSuite were run locally. This contribution is done on behalf of Criteo, according to the terms of the Apache license 2.0. Author: Anthony Truchet <[email protected]> Closes #14268 from AnthonyTruchet/SPARK-16440.
1 parent 75146be commit 0dc79ff

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -434,9 +434,9 @@ class Word2Vec extends Serializable with Logging {
434434
bcSyn1Global.unpersist(false)
435435
}
436436
newSentences.unpersist()
437-
expTable.unpersist()
438-
bcVocab.unpersist()
439-
bcVocabHash.unpersist()
437+
expTable.destroy()
438+
bcVocab.destroy()
439+
bcVocabHash.destroy()
440440

441441
val wordArray = vocab.map(_.word)
442442
new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)

0 commit comments

Comments
 (0)