Skip to content

[SPARK-12630][Python][MLlib][DOC] Update param descriptions in classification.py #10598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-12630][Python][MLlib][DOC] Update param descriptions in classification.py #10598

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e74831f
[SPARK-12630][DOC] Update param descriptions
vijaykiran Jan 5, 2016
6cb46ca
Style fixes
vijaykiran Jan 6, 2016
cbd9d08
Limit parameter desciptions to 74 columns
vijaykiran Jan 22, 2016
bf8f8a0
A couple of more 74 column fixes
vijaykiran Jan 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
261 changes: 143 additions & 118 deletions python/pyspark/mllib/classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,16 +94,19 @@ class LogisticRegressionModel(LinearClassificationModel):
Classification model trained using Multinomial/Binary Logistic
Regression.

:param weights: Weights computed for every feature.
:param intercept: Intercept computed for this model. (Only used
in Binary Logistic Regression. In Multinomial Logistic
Regression, the intercepts will not be a single value,
so the intercepts will be part of the weights.)
:param numFeatures: the dimension of the features.
:param numClasses: the number of possible outcomes for k classes
classification problem in Multinomial Logistic Regression.
By default, it is binary logistic regression so numClasses
will be set to 2.
:param weights:
Weights computed for every feature.
:param intercept:
Intercept computed for this model. (Only used in Binary Logistic
Regression. In Multinomial Logistic Regression, the intercepts will
not bea single value, so the intercepts will be part of the
weights.)
:param numFeatures:
The dimension of the features.
:param numClasses:
The number of possible outcomes for k classes classification problem
in Multinomial Logistic Regression. By default, it is binary
logistic regression so numClasses will be set to 2.

>>> data = [
... LabeledPoint(0.0, [0.0, 1.0]),
Expand Down Expand Up @@ -189,8 +192,8 @@ def numFeatures(self):
@since('1.4.0')
def numClasses(self):
"""
Number of possible outcomes for k classes classification problem in Multinomial
Logistic Regression.
Number of possible outcomes for k classes classification problem
in Multinomial Logistic Regression.
"""
return self._numClasses

Expand Down Expand Up @@ -272,37 +275,42 @@ def train(cls, data, iterations=100, step=1.0, miniBatchFraction=1.0,
"""
Train a logistic regression model on the given data.

:param data: The training data, an RDD of
LabeledPoint.
:param iterations: The number of iterations
(default: 100).
:param step: The step parameter used in SGD
(default: 1.0).
:param miniBatchFraction: Fraction of data to be used for each
SGD iteration (default: 1.0).
:param initialWeights: The initial weights (default: None).
:param regParam: The regularizer parameter
(default: 0.01).
:param regType: The type of regularizer used for
training our model.

:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization

(default: "l2")

:param intercept: Boolean parameter which indicates the
use or not of the augmented representation
for training data (i.e. whether bias
features are activated or not,
default: False).
:param validateData: Boolean parameter which indicates if
the algorithm should validate data
before training. (default: True)
:param convergenceTol: A condition which decides iteration termination.
(default: 0.001)
:param data:
The training data, an RDD of LabeledPoint.
:param iterations:
The number of iterations.
(default: 100)
:param step:
The step parameter used in SGD.
(default: 1.0)
:param miniBatchFraction:
Fraction of data to be used for each SGD iteration.
(default: 1.0)
:param initialWeights:
The initial weights.
(default: None)
:param regParam:
The regularizer parameter.
(default: 0.01)
:param regType:
The type of regularizer used for training our model.
:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization
(default: "l2")
Copy link
Member

@BryanCutler BryanCutler Feb 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we have a problem with the formatting of this param. If we want the html to look the same and get rid of any warnings, then I think it has to be written like this

:param regType:
  The type of regularizer used for training our model.

  :Allowed values:
    - "l1" for using L1 regularization
    - "l2" for using L2 regularization
    - None for no regularization

  (default: "l2")

which, I think, looks ugly in the code and not even that great in the html. Alternatively, we could rewrite it inline similar to how some other params have it, like this

:param regType:
  The type of regularizer used for training our model.
  Allowed values: "l1" for using L1 regularization;
  "l2" for using L2 regularization; None for no
  regularization.
  (default: "l2")

Not perfect, but both code and html look decent, imho.
@mengxr , would you be ok with the second inline format?

Copy link
Contributor

@mengxr mengxr Feb 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first one looks better but no strong preference. I tried the following that also works:

        :param regType:
          The type of regularizer used for training our model.
          Allowed values:

            - "l1" for using L1 regularization
            - "l2" for using L2 regularization (default)
            - None for no regularization

Note Allow values is not a Sphinx keyword.

Copy link
Member

@BryanCutler BryanCutler Feb 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a huge deal so I'm ok with either, but my issue with the additional blank lines is that in the context of the already large param list, the extra blank lines seem like a little much and the rendered html looks sloppy to me too. Since I noticed other parts describing allowed values in a single paragraph, I thought maybe we should stick to that.

Copy link
Contributor

@mengxr mengxr Feb 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we use :Allowed values:, it shows up as :Allowed values: in html, which is weird. We should change existing ones instead of repeating the mistakes, in a separate PR. I don't think the rendered html changes due to that additional blank line. It is just control by top margin of a list, which could be configured via CSS. This is how it looks in html for my proposal, which looks good to me:

screen shot 2016-02-11 at 2 21 41 pm

Let's quickly fix this one and merge this PR.

Copy link
Member

@BryanCutler BryanCutler Feb 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that :Allowed values: would need to be preceded by a blank line for Sphinx to interpret it in bold like before, but I don't think that needs to be done here. What @mengxr proposed above looks fine with me too.

@vijaykiran , will you be able to make this fix soon? I could take over and finish this last little bit if you can't.

:param intercept:
Boolean parameter which indicates the use or not of the
augmented representation for training data (i.e., whether bias
features are activated or not).
(default: False)
:param validateData:
Boolean parameter which indicates if the algorithm should
validate data before training.
(default: True)
:param convergenceTol:
A condition which decides iteration termination.
(default: 0.001)
"""
def train(rdd, i):
return callMLlibFunc("trainLogisticRegressionModelWithSGD", rdd, int(iterations),
Expand All @@ -323,38 +331,43 @@ def train(cls, data, iterations=100, initialWeights=None, regParam=0.01, regType
"""
Train a logistic regression model on the given data.

:param data: The training data, an RDD of
LabeledPoint.
:param iterations: The number of iterations
(default: 100).
:param initialWeights: The initial weights (default: None).
:param regParam: The regularizer parameter
(default: 0.01).
:param regType: The type of regularizer used for
training our model.

:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization

(default: "l2")

:param intercept: Boolean parameter which indicates the
use or not of the augmented representation
for training data (i.e. whether bias
features are activated or not,
default: False).
:param corrections: The number of corrections used in the
LBFGS update (default: 10).
:param tolerance: The convergence tolerance of iterations
for L-BFGS (default: 1e-4).
:param validateData: Boolean parameter which indicates if the
algorithm should validate data before
training. (default: True)
:param numClasses: The number of classes (i.e., outcomes) a
label can take in Multinomial Logistic
Regression (default: 2).
:param data:
The training data, an RDD of LabeledPoint.
:param iterations:
The number of iterations.
(default: 100)
:param initialWeights:
The initial weights.
(default: None)
:param regParam:
The regularizer parameter.
(default: 0.01)
:param regType:
The type of regularizer used for training our model.
:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization
(default: "l2")
:param intercept:
Boolean parameter which indicates the use or not of the
augmented representation for training data (i.e., whether bias
features are activated or not).
(default: False)
:param corrections:
The number of corrections used in the LBFGS update.
(default: 10)
:param tolerance:
The convergence tolerance of iterations for L-BFGS.
(default: 1e-4)
:param validateData:
Boolean parameter which indicates if the algorithm should
validate data before training.
(default: True)
:param numClasses:
The number of classes (i.e., outcomes) a label can take in
Multinomial Logistic Regression.
(default: 2)

>>> data = [
... LabeledPoint(0.0, [0.0, 1.0]),
Expand Down Expand Up @@ -387,8 +400,10 @@ class SVMModel(LinearClassificationModel):
"""
Model for Support Vector Machines (SVMs).

:param weights: Weights computed for every feature.
:param intercept: Intercept computed for this model.
:param weights:
Weights computed for every feature.
:param intercept:
Intercept computed for this model.

>>> data = [
... LabeledPoint(0.0, [0.0]),
Expand Down Expand Up @@ -490,37 +505,42 @@ def train(cls, data, iterations=100, step=1.0, regParam=0.01,
"""
Train a support vector machine on the given data.

:param data: The training data, an RDD of
LabeledPoint.
:param iterations: The number of iterations
(default: 100).
:param step: The step parameter used in SGD
(default: 1.0).
:param regParam: The regularizer parameter
(default: 0.01).
:param miniBatchFraction: Fraction of data to be used for each
SGD iteration (default: 1.0).
:param initialWeights: The initial weights (default: None).
:param regType: The type of regularizer used for
training our model.

:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization

(default: "l2")

:param intercept: Boolean parameter which indicates the
use or not of the augmented representation
for training data (i.e. whether bias
features are activated or not,
default: False).
:param validateData: Boolean parameter which indicates if
the algorithm should validate data
before training. (default: True)
:param convergenceTol: A condition which decides iteration termination.
(default: 0.001)
:param data:
The training data, an RDD of LabeledPoint.
:param iterations:
The number of iterations.
(default: 100)
:param step:
The step parameter used in SGD.
(default: 1.0)
:param regParam:
The regularizer parameter.
(default: 0.01)
:param miniBatchFraction:
Fraction of data to be used for each SGD iteration.
(default: 1.0)
:param initialWeights:
The initial weights.
(default: None)
:param regType:
The type of regularizer used for training our model.
:Allowed values:
- "l1" for using L1 regularization
- "l2" for using L2 regularization
- None for no regularization
(default: "l2")
:param intercept:
Boolean parameter which indicates the use or not of the
augmented representation for training data (i.e. whether bias
features are activated or not).
(default: False)
:param validateData:
Boolean parameter which indicates if the algorithm should
validate data before training.
(default: True)
:param convergenceTol:
A condition which decides iteration termination.
(default: 0.001)
"""
def train(rdd, i):
return callMLlibFunc("trainSVMModelWithSGD", rdd, int(iterations), float(step),
Expand All @@ -536,11 +556,13 @@ class NaiveBayesModel(Saveable, Loader):
"""
Model for Naive Bayes classifiers.

:param labels: list of labels.
:param pi: log of class priors, whose dimension is C,
number of labels.
:param theta: log of class conditional probabilities, whose
dimension is C-by-D, where D is number of features.
:param labels:
List of labels.
:param pi:
Log of class priors, whose dimension is C, number of labels.
:param theta:
Log of class conditional probabilities, whose dimension is C-by-D,
where D is number of features.

>>> data = [
... LabeledPoint(0.0, [0.0, 0.0]),
Expand Down Expand Up @@ -639,8 +661,11 @@ def train(cls, data, lambda_=1.0):
it can also be used as Bernoulli NB (U{http://tinyurl.com/p7c96j6}).
The input feature values must be nonnegative.

:param data: RDD of LabeledPoint.
:param lambda_: The smoothing parameter (default: 1.0).
:param data:
RDD of LabeledPoint.
:param lambda_:
The smoothing parameter.
(default: 1.0)
"""
first = data.first()
if not isinstance(first, LabeledPoint):
Expand All @@ -652,9 +677,9 @@ def train(cls, data, lambda_=1.0):
@inherit_doc
class StreamingLogisticRegressionWithSGD(StreamingLinearAlgorithm):
"""
Train or predict a logistic regression model on streaming data. Training uses
Stochastic Gradient Descent to update the model based on each new batch of
incoming data from a DStream.
Train or predict a logistic regression model on streaming data.
Training uses Stochastic Gradient Descent to update the model based on
each new batch of incoming data from a DStream.

Each batch of data is assumed to be an RDD of LabeledPoints.
The number of data points per batch can vary, but the number
Expand Down