Skip to content

[SPARK-11852] [ML] StandardScaler minor refactor #9839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-11852] [ML] StandardScaler minor refactor #9839

wants to merge 5 commits into from

Conversation

@yanboliang
Copy link
Contributor

@yanboliang yanboliang commented Nov 19, 2015

withStd and withMean should be params of StandardScaler and StandardScalerModel.

yanboliang
yanboliang reviewed Nov 19, 2015
View reviewed changes
mllib/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala Outdated
Copy link
Contributor Author

@yanboliang yanboliang Nov 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need to check mean and std which are parts of the model, withStd and withStd are params.

@yanboliang yanboliang mentioned this pull request Nov 19, 2015
Closed
yanboliang
yanboliang reviewed Nov 19, 2015
View reviewed changes
mllib/src/test/scala/org/apache/spark/ml/feature/StandardScalerSuite.scala Outdated
Copy link
Contributor Author

@yanboliang yanboliang Nov 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

withStd and withStd of StandardScalerModel must be inherited from StandardScaler, so we can not construct StandardScalerModel directly by specifying the two variables. Here we combine the original test cases into one with testEstimatorAndModelReadWrite which both test the estimator and model.

Copy link
Contributor

@mengxr mengxr Nov 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not an ideal unit test for read/write because the model fitting part shouldn't be part of it, which is already covered by other tests. Constructing estimator and model directly can save some test time.

@SparkQA
Copy link

SparkQA commented Nov 19, 2015

Test build #46330 has finished for PR 9839 at commit 37fe45d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 19, 2015

Test build #46333 has finished for PR 9839 at commit 76ef338.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

mengxr
mengxr reviewed Nov 20, 2015
View reviewed changes
mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala Outdated
Copy link
Contributor

@mengxr mengxr Nov 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If withMean and withStd are parameters, we should save them in metadata/ but not both under data/ and medadata/. Can we change the constructor of ml.StandardScalerModel to take only std and mean but construct scaler only inside transform? So scaler is no longer a member variable. We can fix performance issues in 1.7.

@yanboliang
Copy link
Contributor Author

yanboliang commented Nov 20, 2015

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 20, 2015

Test build #46413 has finished for PR 9839 at commit c6b6d7e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 20, 2015

Test build #46415 has finished for PR 9839 at commit c6b6d7e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Nov 20, 2015
@yanboliang @mengxr
[SPARK-11852][ML] StandardScaler minor refactor
b11aa17 
```withStd``` and ```withMean``` should be params of ```StandardScaler``` and ```StandardScalerModel```.

Author: Yanbo Liang <[email protected]>

Closes #9839 from yanboliang/standardScaler-refactor.

(cherry picked from commit 9ace2e5)
Signed-off-by: Xiangrui Meng <[email protected]>
@mengxr
Copy link
Contributor

mengxr commented Nov 20, 2015

LGTM. Merged into master and branch-1.6. Thanks!

@asfgit asfgit closed this in 9ace2e5 Nov 20, 2015
@yanboliang yanboliang deleted the standardScaler-refactor branch November 22, 2015 10:31
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Sep 24, 2024
rdar://133400915 Bump Iceberg to 1.5.0.3-apple (apache#2021)
71236d4 
https://github.pie.apple.com/IPR/apache-incubator-iceberg/compare/IPR:apache-incubator-iceberg:ece9b3117...IPR:apache-incubator-iceberg:85968ae4b

Release notes:

- [Make OAuth audience and resource configurable (apache#9839)](https://github.pie.apple.com/IPR/apache-incubator-iceberg/commit/b204803e560be2dfe58c46a47f262a2be920745a) 
- [Internal: [SPJ] Add bucket to reducible function (](https://github.pie.apple.com/IPR/apache-incubator-iceberg/commit/2d64f72b5728a49e9907d4769497ce5f2b177756)[#1285](https://github.pie.apple.com/IPR/apache-incubator-iceberg/pull/1285)[)](https://github.pie.apple.com/IPR/apache-incubator-iceberg/commit/2d64f72b5728a49e9907d4769497ce5f2b177756)
- Core, Spark: Calling rewrite_position_delete_files fails on tables with more than 1k columns (apache#10020) ([apache#1283](https://github.pie.apple.com/IPR/apache-incubator-iceberg/pull/1283))
- [[Internal] Update revapi version (](https://github.pie.apple.com/IPR/apache-incubator-iceberg/commit/ece9b311752f4f27700ac804f7503658e5de9fab)[#1282](https://github.pie.apple.com/IPR/apache-incubator-iceberg/pull/1282)[)](https://github.pie.apple.com/IPR/apache-incubator-iceberg/commit/ece9b311752f4f27700ac804f7503658e5de9fab)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yanboliang @SparkQA @mengxr