Skip to content

Conversation

@martinbrown
Copy link

Unsure what to do for SQLTransformer since it's in master but not in any tagged release.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also annotate the constructor and the public variables in the constructor, e.g., https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala#L42. Same for other constructors.

@mengxr
Copy link
Contributor

mengxr commented Aug 28, 2015

Looking at the changes, I wonder whether we need to annotate original public methods/variables that have the same since version as the class. Java doesn't do this (http://docs.oracle.com/javase/7/docs/api/java/util/Arrays.html). Please don't remove them now. Just want to discuss and see which approach is more appropriate.

+@rxin since this also applies to DataFrame.

@rxin
Copy link
Contributor

rxin commented Aug 29, 2015

We might as well do it. (Any downside?)

@mengxr
Copy link
Contributor

mengxr commented Aug 30, 2015

The downside would be if we forget since tags in some new methods, it might cause confusions.

@rxin
Copy link
Contributor

rxin commented Aug 31, 2015

I'm for adding them always -- easier to track down which methods don't have them this way.

@mengxr
Copy link
Contributor

mengxr commented Aug 31, 2015

Okay. @martinbrown Could you add @Since to the constructor and public params in it as well? Thanks!

@mengxr
Copy link
Contributor

mengxr commented Sep 16, 2015

ping

@martinbrown
Copy link
Author

Ready for another look

@mengxr
Copy link
Contributor

mengxr commented Sep 17, 2015

ok to test

@mengxr
Copy link
Contributor

mengxr commented Sep 17, 2015

@martinbrown there are some merge conflicts. could you rebase master?

@SparkQA
Copy link

SparkQA commented Sep 17, 2015

Test build #42609 has finished for PR 8505 at commit fc98720.

  • This patch fails Scala style tests.
  • This patch does not merge cleanly.
  • This patch adds the following public classes (experimental):
    • final class Binarizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • final class Bucketizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class CountVectorizerModel @Since("1.5.0") (@Since("1.5.0") override val uid:
    • class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class HashingTF @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Transformer with HasInputCol with HasOutputCol
    • final class IDF @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Estimator[IDFModel] with IDFBase
    • class MinMaxScaler @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends UnaryTransformer[Vector, Vector, Normalizer]
    • class OneHotEncoder @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • class PCA @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[PCAModel] with PCAParams
    • class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class RFormula @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[RFormulaModel] with RFormulaBase
    • class StandardScaler @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Estimator[StandardScalerModel]
    • class StopWordsRemover @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class StringIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Estimator[StringIndexerModel]
    • class Tokenizer @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends UnaryTransformer[String, Seq[String], Tokenizer]
    • class RegexTokenizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorAssembler @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Estimator[VectorIndexerModel]
    • final class VectorSlicer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • final class Word2Vec @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Estimator[Word2VecModel] with Word2VecBase

@SparkQA
Copy link

SparkQA commented Sep 19, 2015

Test build #42707 has started for PR 8505 at commit 31bf4dd.

@martinbrown
Copy link
Author

@mengxr Can you kick off another Jenkins run? It looks like the last run had a fault unrelated to the code.

@SparkQA
Copy link

SparkQA commented Sep 20, 2015

Test build #42720 has finished for PR 8505 at commit d5c6ad8.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • final class Binarizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • final class Bucketizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class CountVectorizerModel @Since("1.5.0") (@Since("1.5.0") override val uid:
    • class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class HashingTF @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • final class IDF @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class MinMaxScaler @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class OneHotEncoder @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • class PCA @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[PCAModel]
    • class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class RFormula @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends
    • class StandardScaler @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class StopWordsRemover @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class StringIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class Tokenizer @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class RegexTokenizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorAssembler @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • final class VectorSlicer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • final class Word2Vec @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends

@SparkQA
Copy link

SparkQA commented Sep 21, 2015

Test build #42779 has finished for PR 8505 at commit d81b8c7.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • final class Binarizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • final class Bucketizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class CountVectorizerModel @Since("1.5.0") (@Since("1.5.0") override val uid:
    • class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class HashingTF @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • final class IDF @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class MinMaxScaler @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class OneHotEncoder @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • class PCA @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[PCAModel]
    • class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class RFormula @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends
    • class StandardScaler @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class StopWordsRemover @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class StringIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class Tokenizer @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class RegexTokenizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorAssembler @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • final class VectorSlicer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • final class Word2Vec @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends

@martinbrown
Copy link
Author

@mengxr any insight on this failure?

@martinbrown
Copy link
Author

Trying again in case it was a transitory download error. Jenkins, retest this please.

@martinbrown
Copy link
Author

@mengxr ping

@SparkQA
Copy link

SparkQA commented Sep 26, 2015

Test build #43040 has finished for PR 8505 at commit aaa2376.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • final class Binarizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • final class Bucketizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class CountVectorizerModel @Since("1.5.0") (@Since("1.5.0") override val uid:
    • class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class HashingTF @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • final class IDF @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class MinMaxScaler @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class OneHotEncoder @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • class PCA @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[PCAModel]
    • class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class RFormula @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends
    • class StandardScaler @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class StopWordsRemover @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class StringIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class Tokenizer @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class RegexTokenizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorAssembler @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • final class VectorSlicer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • final class Word2Vec @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends

@mengxr
Copy link
Contributor

mengxr commented Sep 29, 2015

test this please

@mengxr
Copy link
Contributor

mengxr commented Sep 29, 2015

@martinbrown I think initially it was some Jenkins caching problem, then a flaky test. Let's try again.

@SparkQA
Copy link

SparkQA commented Sep 29, 2015

Test build #43088 has finished for PR 8505 at commit aaa2376.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • final class Binarizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • final class Bucketizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class CountVectorizer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class CountVectorizerModel @Since("1.5.0") (@Since("1.5.0") override val uid:
    • class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class HashingTF @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • final class IDF @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class MinMaxScaler @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class NGram @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class OneHotEncoder @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends Transformer
    • class PCA @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends Estimator[PCAModel]
    • class PolynomialExpansion @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class RFormula @Since("1.5.0") (@Since("1.5.0") override val uid: String) extends
    • class StandardScaler @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class StopWordsRemover @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • class StringIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • class Tokenizer @Since("1.2.0") (@Since("1.4.0") override val uid: String) extends
    • class RegexTokenizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorAssembler @Since("1.4.0") (@Since("1.4.0") override val uid: String)
    • class VectorIndexer @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends
    • final class VectorSlicer @Since("1.5.0") (@Since("1.5.0") override val uid: String)
    • final class Word2Vec @Since("1.4.0") (@Since("1.4.0") override val uid: String) extends

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default constructor should be since 1.2.0. Please check others as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mengxr I'm a little confused here because Binarizer wasn't added until 1.4.0. I did correct the default constructor to 1.2.0 for the original classes from that version (HashingTF, StandardScaler, Tokenizer).

@mengxr
Copy link
Contributor

mengxr commented Sep 30, 2015

LGTM except some minor issues with constructor versions

@mengxr
Copy link
Contributor

mengxr commented Nov 3, 2015

@martinbrown Do you have time to address my comments?

@martinbrown
Copy link
Author

@mengxr responded

@SparkQA
Copy link

SparkQA commented Nov 6, 2015

Test build #45184 has finished for PR 8505 at commit b0881a3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * final class Binarizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * final class Bucketizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class CountVectorizer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class CountVectorizerModel @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid:\n * class DCT @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class ElementwiseProduct @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class HashingTF @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends Transformer\n * final class IDF @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class MinMaxScaler @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class NGram @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class Normalizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class OneHotEncoder @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends Transformer\n * class PCA @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String) extends Estimator[PCAModel]\n * class PolynomialExpansion @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class RFormula @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String) extends\n * class StandardScaler @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class StopWordsRemover @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class StringIndexer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class Tokenizer @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class RegexTokenizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class VectorAssembler @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class VectorIndexer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * final class VectorSlicer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * final class Word2Vec @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * final class DecisionTreeRegressor @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * final class GBTRegressor @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class IsotonicRegression @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class LinearRegression @Since(\"1.3.0\") (@Since(\"1.3.0\") override val uid: String)\n * final class RandomForestRegressor @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n

@SparkQA
Copy link

SparkQA commented Nov 6, 2015

Test build #45189 has finished for PR 8505 at commit b2cc944.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * final class Binarizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * final class Bucketizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class CountVectorizer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class CountVectorizerModel @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid:\n * class DCT @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class ElementwiseProduct @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class HashingTF @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends Transformer\n * final class IDF @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class MinMaxScaler @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class NGram @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class Normalizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class OneHotEncoder @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends Transformer\n * class PCA @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String) extends Estimator[PCAModel]\n * class PolynomialExpansion @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class RFormula @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String) extends\n * class StandardScaler @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class StopWordsRemover @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * class StringIndexer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class Tokenizer @Since(\"1.2.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * class RegexTokenizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class VectorAssembler @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class VectorIndexer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n * final class VectorSlicer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * final class Word2Vec @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends\n

@jkbradley
Copy link
Member

@martinbrown It looks like it's been a while since there has been activity. Sorry! Could you please fix the conflicts, and we can see about merging this for 1.6?

@rxin
Copy link
Contributor

rxin commented Jun 15, 2016

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one.

@asfgit asfgit closed this in 1a33f2e Jun 15, 2016
asfgit pushed a commit that referenced this pull request Jun 21, 2016
This PR adds missing `Since` annotations to `ml.feature` package.

Closes #8505.

## How was this patch tested?

Existing tests.

Author: Nick Pentreath <[email protected]>

Closes #13641 from MLnick/add-since-annotations.
asfgit pushed a commit that referenced this pull request Jun 21, 2016
This PR adds missing `Since` annotations to `ml.feature` package.

Closes #8505.

## How was this patch tested?

Existing tests.

Author: Nick Pentreath <[email protected]>

Closes #13641 from MLnick/add-since-annotations.

(cherry picked from commit 37494a1)
Signed-off-by: Xiangrui Meng <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants