Skip to content

[SPARK-20736][Python] PySpark StringIndexer supports StringOrderType #17978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

[SPARK-20736][Python] PySpark StringIndexer supports StringOrderType #17978

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ddf34a5
Python API to StringOrderType in StringIndexer
May 14, 2017
c1966bb
fix typo
May 14, 2017
e5c8dcf
fix typo
May 14, 2017
bd80b37
fix style
May 14, 2017
1f336ab
fix style
May 14, 2017
44f0a36
add tests
May 15, 2017
f66a445
fix test error
May 15, 2017
36006bf
address comments
May 16, 2017
6acabc2
minor style fix
May 16, 2017
2fe9432
add default value for stringOrderType in docstring
May 20, 2017
5bfa4dc
fix example error
May 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 43 additions & 8 deletions python/pyspark/ml/feature.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2082,10 +2082,12 @@ class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, HasHandleInvalid,
"""
A label indexer that maps a string column of labels to an ML column of label indices.
If the input column is numeric, we cast it to string and index the string values.
The indices are in [0, numLabels), ordered by label frequencies.
So the most frequent label gets index 0.
The indices are in [0, numLabels). By default, this is ordered by label frequencies
so the most frequent label gets index 0. The ordering behavior is controlled by
setting :py:attr:`stringOrderType`. Its default value is 'frequencyDesc'.

>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed", handleInvalid='error')
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed", handleInvalid="error",
... stringOrderType="frequencyDesc")
>>> model = stringIndexer.fit(stringIndDf)
>>> td = model.transform(stringIndDf)
>>> sorted(set([(i[0], i[1]) for i in td.select(td.id, td.indexed).collect()]),
Expand All @@ -2111,26 +2113,45 @@ class StringIndexer(JavaEstimator, HasInputCol, HasOutputCol, HasHandleInvalid,
>>> loadedInverter = IndexToString.load(indexToStringPath)
>>> loadedInverter.getLabels() == inverter.getLabels()
True
>>> stringIndexer.getStringOrderType()
'frequencyDesc'
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed", handleInvalid="error",
... stringOrderType="alphabetDesc")
>>> model = stringIndexer.fit(stringIndDf)
>>> td = model.transform(stringIndDf)
>>> sorted(set([(i[0], i[1]) for i in td.select(td.id, td.indexed).collect()]),
... key=lambda x: x[0])
[(0, 2.0), (1, 1.0), (2, 0.0), (3, 2.0), (4, 2.0), (5, 0.0)]

.. versionadded:: 1.4.0
"""

stringOrderType = Param(Params._dummy(), "stringOrderType",
"How to order labels of string column. The first label after " +
"ordering is assigned an index of 0. Supported options: " +
"frequencyDesc, frequencyAsc, alphabetDesc, alphabetAsc.",
Copy link
Member

@felixcheung felixcheung May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be generated instead of hardcoded - you can find a example on python..

Copy link
Contributor Author

@actuaryzhang actuaryzhang May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixcheung stringOrderType is not a shared trait on the Scala side, and I thought only the shared traits should be automatically generated.
I have looked at the code for other ML transformers, and many of them hard coded, for example, Imputer and OneHotEncoder.
Please let me know if I'm wrong, and a reference to example would be greatly appreciated. Thanks much!

Copy link
Member

@felixcheung felixcheung May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, ok, I see a few examples that they are not generated

Copy link
Contributor

@holdenk holdenk May 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know were mixed on doing this, but I like including the default value in the docstring, makes the documentation closer to the Scala doc and makes it easier to read without having to refer to the ScalaDoc.

typeConverter=TypeConverters.toString)

@keyword_only
def __init__(self, inputCol=None, outputCol=None, handleInvalid="error"):
def __init__(self, inputCol=None, outputCol=None, handleInvalid="error",
stringOrderType="frequencyDesc"):
"""
__init__(self, inputCol=None, outputCol=None, handleInvalid="error")
__init__(self, inputCol=None, outputCol=None, handleInvalid="error", \
Copy link
Member

@HyukjinKwon HyukjinKwon May 15, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we need at least a doctest.

Copy link
Contributor Author

@actuaryzhang actuaryzhang May 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HyukjinKwon Thank you. Added tests.

stringOrderType="frequencyDesc")
"""
super(StringIndexer, self).__init__()
self._java_obj = self._new_java_obj("org.apache.spark.ml.feature.StringIndexer", self.uid)
self._setDefault(handleInvalid="error")
self._setDefault(handleInvalid="error", stringOrderType="frequencyDesc")
kwargs = self._input_kwargs
self.setParams(**kwargs)

@keyword_only
@since("1.4.0")
def setParams(self, inputCol=None, outputCol=None, handleInvalid="error"):
def setParams(self, inputCol=None, outputCol=None, handleInvalid="error",
stringOrderType="frequencyDesc"):
"""
setParams(self, inputCol=None, outputCol=None, handleInvalid="error")
setParams(self, inputCol=None, outputCol=None, handleInvalid="error", \
stringOrderType="frequencyDesc")
Sets params for this StringIndexer.
"""
kwargs = self._input_kwargs
Expand All @@ -2139,6 +2160,20 @@ def setParams(self, inputCol=None, outputCol=None, handleInvalid="error"):
def _create_model(self, java_model):
return StringIndexerModel(java_model)

@since("2.3.0")
def setStringOrderType(self, value):
"""
Sets the value of :py:attr:`stringOrderType`.
"""
return self._set(stringOrderType=value)

@since("2.3.0")
def getStringOrderType(self):
"""
Gets the value of :py:attr:`stringOrderType` or its default value 'frequencyDesc'.
"""
return self.getOrDefault(self.stringOrderType)


class StringIndexerModel(JavaModel, JavaMLReadable, JavaMLWritable):
"""
Expand Down