Skip to content

Commit 8765665

Browse files
hhbyyhmengxr
hhbyyh
authored and
mengxr
committed
[SPARK-8169] [ML] Add StopWordsRemover as a transformer
jira: https://issues.apache.org/jira/browse/SPARK-8169 stop words: http://en.wikipedia.org/wiki/Stop_words StopWordsRemover takes a string array column and outputs a string array column with all defined stop words removed. The transformer should also come with a standard set of stop words as default. Currently I used a minimum stop words set since on some [case](http://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html), small set of stop words is preferred. ASCII char has been tested, Yet I cannot check it in due to style check. Further thought, 1. Maybe I should use OpenHashSet. Is it recommended? 2. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null). 3. If the current stop words set looks too limited, any suggestion for replacement? We can have something similar to the one in [SKlearn](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py). Author: Yuhao Yang <[email protected]> Closes apache#6742 from hhbyyh/stopwords and squashes the following commits: fa959d8 [Yuhao Yang] separating udf f190217 [Yuhao Yang] replace default list and other small fix 04403ab [Yuhao Yang] Merge remote-tracking branch 'upstream/master' into stopwords b3aa957 [Yuhao Yang] add stopWordsRemover
1 parent d2a9b66 commit 8765665

File tree

2 files changed

+235
-0
lines changed

2 files changed

+235
-0
lines changed

‎mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.ml.feature
19+
20+
import org.apache.spark.annotation.Experimental
21+
import org.apache.spark.ml.Transformer
22+
import org.apache.spark.ml.param.shared.{HasInputCol, HasOutputCol}
23+
import org.apache.spark.ml.param.{ParamMap, BooleanParam, Param}
24+
import org.apache.spark.ml.util.Identifiable
25+
import org.apache.spark.sql.DataFrame
26+
import org.apache.spark.sql.types.{StringType, StructField, ArrayType, StructType}
27+
import org.apache.spark.sql.functions.{col, udf}
28+
29+
/**
30+
* stop words list
31+
*/
32+
private object StopWords {
33+
34+
/**
35+
* Use the same default stopwords list as scikit-learn.
36+
* The original list can be found from "Glasgow Information Retrieval Group"
37+
* [[http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words]]
38+
*/
39+
val EnglishStopWords = Array( "a", "about", "above", "across", "after", "afterwards", "again",
40+
"against", "all", "almost", "alone", "along", "already", "also", "although", "always",
41+
"am", "among", "amongst", "amoungst", "amount", "an", "and", "another",
42+
"any", "anyhow", "anyone", "anything", "anyway", "anywhere", "are",
43+
"around", "as", "at", "back", "be", "became", "because", "become",
44+
"becomes", "becoming", "been", "before", "beforehand", "behind", "being",
45+
"below", "beside", "besides", "between", "beyond", "bill", "both",
46+
"bottom", "but", "by", "call", "can", "cannot", "cant", "co", "con",
47+
"could", "couldnt", "cry", "de", "describe", "detail", "do", "done",
48+
"down", "due", "during", "each", "eg", "eight", "either", "eleven", "else",
49+
"elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone",
50+
"everything", "everywhere", "except", "few", "fifteen", "fify", "fill",
51+
"find", "fire", "first", "five", "for", "former", "formerly", "forty",
52+
"found", "four", "from", "front", "full", "further", "get", "give", "go",
53+
"had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter",
54+
"hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his",
55+
"how", "however", "hundred", "i", "ie", "if", "in", "inc", "indeed",
56+
"interest", "into", "is", "it", "its", "itself", "keep", "last", "latter",
57+
"latterly", "least", "less", "ltd", "made", "many", "may", "me",
58+
"meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly",
59+
"move", "much", "must", "my", "myself", "name", "namely", "neither",
60+
"never", "nevertheless", "next", "nine", "no", "nobody", "none", "noone",
61+
"nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on",
62+
"once", "one", "only", "onto", "or", "other", "others", "otherwise", "our",
63+
"ours", "ourselves", "out", "over", "own", "part", "per", "perhaps",
64+
"please", "put", "rather", "re", "same", "see", "seem", "seemed",
65+
"seeming", "seems", "serious", "several", "she", "should", "show", "side",
66+
"since", "sincere", "six", "sixty", "so", "some", "somehow", "someone",
67+
"something", "sometime", "sometimes", "somewhere", "still", "such",
68+
"system", "take", "ten", "than", "that", "the", "their", "them",
69+
"themselves", "then", "thence", "there", "thereafter", "thereby",
70+
"therefore", "therein", "thereupon", "these", "they", "thick", "thin",
71+
"third", "this", "those", "though", "three", "through", "throughout",
72+
"thru", "thus", "to", "together", "too", "top", "toward", "towards",
73+
"twelve", "twenty", "two", "un", "under", "until", "up", "upon", "us",
74+
"very", "via", "was", "we", "well", "were", "what", "whatever", "when",
75+
"whence", "whenever", "where", "whereafter", "whereas", "whereby",
76+
"wherein", "whereupon", "wherever", "whether", "which", "while", "whither",
77+
"who", "whoever", "whole", "whom", "whose", "why", "will", "with",
78+
"within", "without", "would", "yet", "you", "your", "yours", "yourself", "yourselves")
79+
}
80+
81+
/**
82+
* :: Experimental ::
83+
* A feature transformer that filters out stop words from input.
84+
* Note: null values from input array are preserved unless adding null to stopWords explicitly.
85+
* @see [[http://en.wikipedia.org/wiki/Stop_words]]
86+
*/
87+
@Experimental
88+
class StopWordsRemover(override val uid: String)
89+
extends Transformer with HasInputCol with HasOutputCol {
90+
91+
def this() = this(Identifiable.randomUID("stopWords"))
92+
93+
/** @group setParam */
94+
def setInputCol(value: String): this.type = set(inputCol, value)
95+
96+
/** @group setParam */
97+
def setOutputCol(value: String): this.type = set(outputCol, value)
98+
99+
/**
100+
* the stop words set to be filtered out
101+
* @group param
102+
*/
103+
val stopWords: Param[Array[String]] = new Param(this, "stopWords", "stop words")
104+
105+
/** @group setParam */
106+
def setStopWords(value: Array[String]): this.type = set(stopWords, value)
107+
108+
/** @group getParam */
109+
def getStopWords: Array[String] = $(stopWords)
110+
111+
/**
112+
* whether to do a case sensitive comparison over the stop words
113+
* @group param
114+
*/
115+
val caseSensitive: BooleanParam = new BooleanParam(this, "caseSensitive",
116+
"whether to do case-sensitive comparison during filtering")
117+
118+
/** @group setParam */
119+
def setCaseSensitive(value: Boolean): this.type = set(caseSensitive, value)
120+
121+
/** @group getParam */
122+
def getCaseSensitive: Boolean = $(caseSensitive)
123+
124+
setDefault(stopWords -> StopWords.EnglishStopWords, caseSensitive -> false)
125+
126+
override def transform(dataset: DataFrame): DataFrame = {
127+
val outputSchema = transformSchema(dataset.schema)
128+
val t = if ($(caseSensitive)) {
129+
val stopWordsSet = $(stopWords).toSet
130+
udf { terms: Seq[String] =>
131+
terms.filter(s => !stopWordsSet.contains(s))
132+
}
133+
} else {
134+
val toLower = (s: String) => if (s != null) s.toLowerCase else s
135+
val lowerStopWords = $(stopWords).map(toLower(_)).toSet
136+
udf { terms: Seq[String] =>
137+
terms.filter(s => !lowerStopWords.contains(toLower(s)))
138+
}
139+
}
140+
141+
val metadata = outputSchema($(outputCol)).metadata
142+
dataset.select(col("*"), t(col($(inputCol))).as($(outputCol), metadata))
143+
}
144+
145+
override def transformSchema(schema: StructType): StructType = {
146+
val inputType = schema($(inputCol)).dataType
147+
require(inputType.sameType(ArrayType(StringType)),
148+
s"Input type must be ArrayType(StringType) but got $inputType.")
149+
val outputFields = schema.fields :+
150+
StructField($(outputCol), inputType, schema($(inputCol)).nullable)
151+
StructType(outputFields)
152+
}
153+
154+
override def copy(extra: ParamMap): StopWordsRemover = defaultCopy(extra)
155+
}

‎mllib/src/test/scala/org/apache/spark/ml/feature/StopWordsRemoverSuite.scala‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.ml.feature
19+
20+
import org.apache.spark.SparkFunSuite
21+
import org.apache.spark.mllib.util.MLlibTestSparkContext
22+
import org.apache.spark.sql.{DataFrame, Row}
23+
24+
object StopWordsRemoverSuite extends SparkFunSuite {
25+
def testStopWordsRemover(t: StopWordsRemover, dataset: DataFrame): Unit = {
26+
t.transform(dataset)
27+
.select("filtered", "expected")
28+
.collect()
29+
.foreach { case Row(tokens, wantedTokens) =>
30+
assert(tokens === wantedTokens)
31+
}
32+
}
33+
}
34+
35+
class StopWordsRemoverSuite extends SparkFunSuite with MLlibTestSparkContext {
36+
import StopWordsRemoverSuite._
37+
38+
test("StopWordsRemover default") {
39+
val remover = new StopWordsRemover()
40+
.setInputCol("raw")
41+
.setOutputCol("filtered")
42+
val dataSet = sqlContext.createDataFrame(Seq(
43+
(Seq("test", "test"), Seq("test", "test")),
44+
(Seq("a", "b", "c", "d"), Seq("b", "c", "d")),
45+
(Seq("a", "the", "an"), Seq()),
46+
(Seq("A", "The", "AN"), Seq()),
47+
(Seq(null), Seq(null)),
48+
(Seq(), Seq())
49+
)).toDF("raw", "expected")
50+
51+
testStopWordsRemover(remover, dataSet)
52+
}
53+
54+
test("StopWordsRemover case sensitive") {
55+
val remover = new StopWordsRemover()
56+
.setInputCol("raw")
57+
.setOutputCol("filtered")
58+
.setCaseSensitive(true)
59+
val dataSet = sqlContext.createDataFrame(Seq(
60+
(Seq("A"), Seq("A")),
61+
(Seq("The", "the"), Seq("The"))
62+
)).toDF("raw", "expected")
63+
64+
testStopWordsRemover(remover, dataSet)
65+
}
66+
67+
test("StopWordsRemover with additional words") {
68+
val stopWords = StopWords.EnglishStopWords ++ Array("python", "scala")
69+
val remover = new StopWordsRemover()
70+
.setInputCol("raw")
71+
.setOutputCol("filtered")
72+
.setStopWords(stopWords)
73+
val dataSet = sqlContext.createDataFrame(Seq(
74+
(Seq("python", "scala", "a"), Seq()),
75+
(Seq("Python", "Scala", "swift"), Seq("swift"))
76+
)).toDF("raw", "expected")
77+
78+
testStopWordsRemover(remover, dataSet)
79+
}
80+
}

0 commit comments

Comments
 (0)