Skip to content

Added setMinCount to Word2Vec.scala #3693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

Added setMinCount to Word2Vec.scala #3693

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5eb9100
Added setMinCount to Word2Vec.scala
ganonp Dec 14, 2014
ffb88bb
Update Word2Vec.scala
ganonp Dec 16, 2014
76bdf5a
Update Word2Vec.scala
ganonp Dec 16, 2014
12ed8f9
Update Word2Vec.scala
ganonp Dec 18, 2014
854958b
Fixed Indentation for setMinCount
ganonp Dec 18, 2014
5110a6f
Reorganized
ganonp Dec 19, 2014
ad534f2
made norm method public
ganonp Dec 20, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ class Word2Vec extends Serializable with Logging {
private var numPartitions = 1
private var numIterations = 1
private var seed = Utils.random.nextLong()

private var minCount = 5

/**
* Sets vector size (default: 100).
*/
Expand Down Expand Up @@ -114,6 +115,15 @@ class Word2Vec extends Serializable with Logging {
this
}

/**
* Sets minCount, the minimum number of times a token must appear to be included in the word2vec
* model's vocabulary (default: 5).
*/
def setMinCount(minCount: Int): this.type = {
this.minCount = minCount
this
}

private val EXP_TABLE_SIZE = 1000
private val MAX_EXP = 6
private val MAX_CODE_LENGTH = 40
Expand All @@ -122,9 +132,6 @@ class Word2Vec extends Serializable with Logging {
/** context words from [-window, window] */
private val window = 5

/** minimum frequency to consider a vocabulary word */
private val minCount = 5

private var trainWordsCount = 0
private var vocabSize = 0
private var vocab: Array[VocabWord] = null
Expand Down
2 changes: 1 addition & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ object Vectors {
* @param p norm.
* @return norm in L^p^ space.
*/
private[spark] def norm(vector: Vector, p: Double): Double = {
def norm(vector: Vector, p: Double): Double = {
require(p >= 1.0)
val values = vector match {
case dv: DenseVector => dv.values
Expand Down