[SPARK-1752][MLLIB] Standardize text format for vectors and labeled points #685

mengxr · 2014-05-07T22:00:13Z

We should standardize the text format used to represent vectors and labeled points. The proposed formats are the following:

dense vector: [v0,v1,..]
sparse vector: (size,[i0,i1],[v0,v1])
labeled point: (label,vector)

where "(..)" indicates a tuple and "[...]" indicate an array. loadLabeledPoints is added to pyspark's MLUtils. I didn't add loadVectors to pyspark because RDD.saveAsTextFile cannot stringify dense vectors in the proposed format automatically.

MLUtils#saveLabeledData and MLUtils#loadLabeledData are deprecated. Users should use RDD#saveAsTextFile and MLUtils#loadLabeledPoints instead. In Scala, MLUtils#loadLabeledPoints is compatible with the format used by MLUtils#loadLabeledData.

CC: @mateiz, @srowen

….0 and deprecate loadLabeledData and saveLabeledData

AmplabJenkins · 2014-05-07T22:02:57Z

Merged build triggered.

AmplabJenkins · 2014-05-07T22:03:05Z

Merged build started.

AmplabJenkins · 2014-05-07T22:40:33Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-07T22:40:33Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14788/

AmplabJenkins · 2014-05-08T00:27:57Z

Merged build triggered.

AmplabJenkins · 2014-05-08T00:28:06Z

Merged build started.

AmplabJenkins · 2014-05-08T00:37:57Z

Merged build triggered.

AmplabJenkins · 2014-05-08T00:38:06Z

Merged build started.

AmplabJenkins · 2014-05-08T01:03:01Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-08T01:03:01Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14795/

AmplabJenkins · 2014-05-08T01:12:23Z

Merged build finished.

AmplabJenkins · 2014-05-08T01:12:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14796/

mateiz · 2014-05-08T01:38:23Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala

Parser combinators are really slow unfortunately, should use something else here

AmplabJenkins · 2014-05-09T00:47:57Z

Merged build triggered.

AmplabJenkins · 2014-05-09T00:48:04Z

Merged build started.

AmplabJenkins · 2014-05-09T01:27:22Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-09T01:27:22Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14831/

mengxr · 2014-06-02T20:15:31Z

There is a saveAsLibSVMFile function in MLUtils.

AmplabJenkins · 2014-06-02T20:17:58Z

Merged build triggered.

AmplabJenkins · 2014-06-02T20:18:04Z

Merged build started.

AmplabJenkins · 2014-06-02T20:58:57Z

Merged build finished.

AmplabJenkins · 2014-06-02T20:58:57Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15354/

…ary compatibility

AmplabJenkins · 2014-06-03T19:47:58Z

Merged build triggered.

AmplabJenkins · 2014-06-03T19:48:06Z

Merged build started.

AmplabJenkins · 2014-06-03T20:28:57Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-03T20:28:57Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15397/

AmplabJenkins · 2014-06-03T21:42:59Z

Merged build triggered.

AmplabJenkins · 2014-06-03T21:43:09Z

Merged build started.

AmplabJenkins · 2014-06-03T22:24:12Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-03T22:24:12Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15409/

mengxr · 2014-06-04T09:06:09Z

@mateiz , is it okay to merge?

mateiz · 2014-06-04T19:57:10Z

Looks good! I've merged it into master.

@mateiz

…oints We should standardize the text format used to represent vectors and labeled points. The proposed formats are the following: 1. dense vector: `[v0,v1,..]` 2. sparse vector: `(size,[i0,i1],[v0,v1])` 3. labeled point: `(label,vector)` where "(..)" indicates a tuple and "[...]" indicate an array. `loadLabeledPoints` is added to pyspark's `MLUtils`. I didn't add `loadVectors` to pyspark because `RDD.saveAsTextFile` cannot stringify dense vectors in the proposed format automatically. `MLUtils#saveLabeledData` and `MLUtils#loadLabeledData` are deprecated. Users should use `RDD#saveAsTextFile` and `MLUtils#loadLabeledPoints` instead. In Scala, `MLUtils#loadLabeledPoints` is compatible with the format used by `MLUtils#loadLabeledData`. CC: @mateiz, @srowen Author: Xiangrui Meng <[email protected]> Closes apache#685 from mengxr/labeled-io and squashes the following commits: 2d1116a [Xiangrui Meng] make loadLabeledData/saveLabeledData deprecated since 1.0.1 297be75 [Xiangrui Meng] change LabeledPoint.parse to LabeledPointParser.parse to maintain binary compatibility d6b1473 [Xiangrui Meng] Merge branch 'master' into labeled-io 56746ea [Xiangrui Meng] replace # by . 623a5f0 [Xiangrui Meng] merge master f06d5ba [Xiangrui Meng] add docs and minor updates 640fe0c [Xiangrui Meng] throw SparkException 5bcfbc4 [Xiangrui Meng] update test to add scientific notations e86bf38 [Xiangrui Meng] remove NumericTokenizer 050fca4 [Xiangrui Meng] use StringTokenizer 6155b75 [Xiangrui Meng] merge master f644438 [Xiangrui Meng] remove parse methods based on eval from pyspark a41675a [Xiangrui Meng] python loadLabeledPoint uses Scala's implementation ce9a475 [Xiangrui Meng] add deserialize_labeled_point to pyspark with tests e9fcd49 [Xiangrui Meng] add serializeLabeledPoint and tests aea4ae3 [Xiangrui Meng] minor updates 810d6df [Xiangrui Meng] update tokenizer/parser implementation 7aac03a [Xiangrui Meng] remove Scala parsers c1885c1 [Xiangrui Meng] add headers and minor changes b0c50cb [Xiangrui Meng] add customized parser d731817 [Xiangrui Meng] style update 63dc396 [Xiangrui Meng] add loadLabeledPoints to pyspark ea122b5 [Xiangrui Meng] Merge branch 'master' into labeled-io cd6c78f [Xiangrui Meng] add __str__ and parse to LabeledPoint a7a178e [Xiangrui Meng] add stringify to pyspark's Vectors 5c2dbfa [Xiangrui Meng] add parse to pyspark's Vectors 7853f88 [Xiangrui Meng] update pyspark's SparseVector.__str__ e761d32 [Xiangrui Meng] make LabelPoint.parse compatible with the dense format used before v1.0 and deprecate loadLabeledData and saveLabeledData 9e63a02 [Xiangrui Meng] add loadVectors and loadLabeledPoints 19aa523 [Xiangrui Meng] update toString and add parsers for Vectors and LabeledPoint

@mateiz

…oints We should standardize the text format used to represent vectors and labeled points. The proposed formats are the following: 1. dense vector: `[v0,v1,..]` 2. sparse vector: `(size,[i0,i1],[v0,v1])` 3. labeled point: `(label,vector)` where "(..)" indicates a tuple and "[...]" indicate an array. `loadLabeledPoints` is added to pyspark's `MLUtils`. I didn't add `loadVectors` to pyspark because `RDD.saveAsTextFile` cannot stringify dense vectors in the proposed format automatically. `MLUtils#saveLabeledData` and `MLUtils#loadLabeledData` are deprecated. Users should use `RDD#saveAsTextFile` and `MLUtils#loadLabeledPoints` instead. In Scala, `MLUtils#loadLabeledPoints` is compatible with the format used by `MLUtils#loadLabeledData`. CC: @mateiz, @srowen Author: Xiangrui Meng <[email protected]> Closes apache#685 from mengxr/labeled-io and squashes the following commits: 2d1116a [Xiangrui Meng] make loadLabeledData/saveLabeledData deprecated since 1.0.1 297be75 [Xiangrui Meng] change LabeledPoint.parse to LabeledPointParser.parse to maintain binary compatibility d6b1473 [Xiangrui Meng] Merge branch 'master' into labeled-io 56746ea [Xiangrui Meng] replace # by . 623a5f0 [Xiangrui Meng] merge master f06d5ba [Xiangrui Meng] add docs and minor updates 640fe0c [Xiangrui Meng] throw SparkException 5bcfbc4 [Xiangrui Meng] update test to add scientific notations e86bf38 [Xiangrui Meng] remove NumericTokenizer 050fca4 [Xiangrui Meng] use StringTokenizer 6155b75 [Xiangrui Meng] merge master f644438 [Xiangrui Meng] remove parse methods based on eval from pyspark a41675a [Xiangrui Meng] python loadLabeledPoint uses Scala's implementation ce9a475 [Xiangrui Meng] add deserialize_labeled_point to pyspark with tests e9fcd49 [Xiangrui Meng] add serializeLabeledPoint and tests aea4ae3 [Xiangrui Meng] minor updates 810d6df [Xiangrui Meng] update tokenizer/parser implementation 7aac03a [Xiangrui Meng] remove Scala parsers c1885c1 [Xiangrui Meng] add headers and minor changes b0c50cb [Xiangrui Meng] add customized parser d731817 [Xiangrui Meng] style update 63dc396 [Xiangrui Meng] add loadLabeledPoints to pyspark ea122b5 [Xiangrui Meng] Merge branch 'master' into labeled-io cd6c78f [Xiangrui Meng] add __str__ and parse to LabeledPoint a7a178e [Xiangrui Meng] add stringify to pyspark's Vectors 5c2dbfa [Xiangrui Meng] add parse to pyspark's Vectors 7853f88 [Xiangrui Meng] update pyspark's SparseVector.__str__ e761d32 [Xiangrui Meng] make LabelPoint.parse compatible with the dense format used before v1.0 and deprecate loadLabeledData and saveLabeledData 9e63a02 [Xiangrui Meng] add loadVectors and loadLabeledPoints 19aa523 [Xiangrui Meng] update toString and add parsers for Vectors and LabeledPoint

* Downgrade R to 3.4.3 * Upgrade Python2 version to fix libffi-related pip install failure * Bump docker image version * Provide version for base docker when building R / python images * Make pyenv work with latest conda * Install R dependencies using conda

Co-authored-by: Egor Krivokon <>

…version 1.1.10.4 (apache#685)

Co-authored-by: Egor Krivokon <>

…istence (apache#685)

mengxr added 3 commits May 7, 2014 13:22

update toString and add parsers for Vectors and LabeledPoint

19aa523

add loadVectors and loadLabeledPoints

9e63a02

make LabelPoint.parse compatible with the dense format used before v1…

e761d32

….0 and deprecate loadLabeledData and saveLabeledData

mengxr added 3 commits May 7, 2014 15:13

update pyspark's SparseVector.__str__

7853f88

add parse to pyspark's Vectors

5c2dbfa

add stringify to pyspark's Vectors

a7a178e

mengxr added 3 commits May 7, 2014 16:03

add __str__ and parse to LabeledPoint

cd6c78f

Merge branch 'master' into labeled-io

ea122b5

add loadLabeledPoints to pyspark

63dc396

style update

d731817

mateiz reviewed May 8, 2014
View reviewed changes

mengxr added 3 commits May 8, 2014 16:21

add customized parser

b0c50cb

add headers and minor changes

c1885c1

remove Scala parsers

7aac03a

mengxr added 2 commits June 2, 2014 16:47

Merge branch 'master' into labeled-io

d6b1473

change LabeledPoint.parse to LabeledPointParser.parse to maintain bin…

297be75

…ary compatibility

make loadLabeledData/saveLabeledData deprecated since 1.0.1

2d1116a

asfgit closed this in 189df16 Jun 4, 2014

davies mentioned this pull request Jun 18, 2015

[SPARK-8291] [MLlib] [PySpark] Add parse functionality to LabeledPoint in PySpark #6746

Closed

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-752] Spark 2.4.5 build hangs with JDK11 (apache#685)

0f45482

Co-authored-by: Egor Krivokon <>

RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Dec 8, 2023

KE-42783 Fix vulnerability, Upgrade org.xerial.snappy:snappy-java to …

d527eef

…version 1.1.10.4 (apache#685)

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-752] Spark 2.4.5 build hangs with JDK11 (apache#685)

cc63412

Co-authored-by: Egor Krivokon <>

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

MapR [SPARK-752] Spark 2.4.5 build hangs with JDK11 (apache#685)

0706bac

Co-authored-by: Egor Krivokon <>

turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025

[HADP-56332][FOLLOWUP] Add more log to track spark-monitor event pers…

c5e34d0

…istence (apache#685)

[SPARK-1752][MLLIB] Standardize text format for vectors and labeled points #685

[SPARK-1752][MLLIB] Standardize text format for vectors and labeled points #685

Uh oh!

Conversation

mengxr commented May 7, 2014

Uh oh!

AmplabJenkins commented May 7, 2014

Uh oh!

AmplabJenkins commented May 7, 2014

Uh oh!

AmplabJenkins commented May 7, 2014

Uh oh!

AmplabJenkins commented May 7, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

mateiz May 8, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

mengxr commented Jun 2, 2014

Uh oh!

AmplabJenkins commented Jun 2, 2014

Uh oh!

AmplabJenkins commented Jun 2, 2014

Uh oh!

AmplabJenkins commented Jun 2, 2014

Uh oh!

AmplabJenkins commented Jun 2, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

AmplabJenkins commented Jun 3, 2014

Uh oh!

mengxr commented Jun 4, 2014

Uh oh!

mateiz commented Jun 4, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants