[SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide #22675

WeichenXu123 · 2018-10-09T02:59:12Z

What changes were proposed in this pull request?

Spark datasource for image/libsvm user guide

How was this patch tested?

Scala:

Java:

Python:

R:

SparkQA · 2018-10-09T03:19:02Z

Test build #97141 has finished for PR 22675 at commit 887f528.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-10-09T03:37:05Z

docs/ml-datasource.md

Out of curiosity, why did we put the image source inside of Spark, rather then a separate module? (see also #21742 (comment)). Avro was put as a separate module.

cc @mengxr as well

Usually it depends on how important the use case is. For example, CSV was created as an external data source and later merged into Spark. See https://issues.apache.org/jira/browse/SPARK-21866?focusedCommentId=16148268&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16148268.

I meant (external) Avro was merged into external/... in Apache Spark as a separate module due to the reason above. Image data source is merged into Spark's main code rather then a separate module. I don't object to bring an external into Apache Spark and I don't doubt you guys's judgement - ++1 for bring this in actually.

My point is I was wondering why this exists in Spark's main code whereas the ideal approach is to put them external/....

cc @cloud-fan and @gatorsmile, am I missing something?

I sympathize with the comment, but I think it makes some sense tucked into ML rather than a standalone module.

felixcheung · 2018-10-09T06:09:11Z

we need this for 2.4?

WeichenXu123 · 2018-10-09T08:33:49Z

This do not block 2.4 release. But merge before 2.4 is better.

srowen

Needs some proofreading but the idea seems OK

srowen · 2018-10-09T14:16:48Z

docs/ml-datasource.md

data sources -> data sources like

Do you want to just say "Parquet, CSV, JSON, and JDBC"? they aren't code identifiers here

parquat -> parquet

data source -> data sources

srowen · 2018-10-09T14:17:39Z

docs/ml-datasource.md

from directory -> from a directory
Maybe say "This data source loads images in libsvm format from a directory"?

srowen · 2018-10-09T14:18:22Z

docs/ml-datasource.md

implements Spark -> implements a Spark
as DataFrame -> as a DataFrame
This sentence is repeated three times. Can you move the shared text out of the language-specific code blocks?

srowen · 2018-10-09T14:19:20Z

docs/ml-datasource.md

scala -> Scala, but this is about Python.

SparkQA · 2018-10-10T05:13:31Z

Test build #97184 has finished for PR 22675 at commit cc4c967.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-10T05:18:46Z

Test build #97185 has finished for PR 22675 at commit 994d09d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-10-11T05:50:27Z

docs/ml-datasource.md

JSON, JDBC -> JSON and JDBC

HyukjinKwon · 2018-10-11T05:53:21Z

docs/ml-datasource.md

Shall we describe which image we can load? For instance, I think this delegates to ImageIO in Java which allows to read compressed format like PNG or JPG to raw image representation like BMP so that OpenCV can handles them.

HyukjinKwon · 2018-10-11T05:53:54Z

docs/ml-datasource.md

I would also describe the schema structure and what each field means.

HyukjinKwon · 2018-10-11T05:55:44Z

docs/ml-datasource.md

how about SQL syntax? I think we can use CREATE TABLE tableA USING LOCATION 'data/image.png'

This looks like SQL features and fit all datasources. Put it in spark SQL doc will be better.

Shall we add an example for R as well then? It wouldn't be too difficult to add the equivalent examples. Also, I don't think we will add the equivalent examples in different languages at different pages.

HyukjinKwon · 2018-10-11T05:56:53Z

docs/ml-datasource.md

Can we do a simple transformation so that how the image datasource can be utilized?

SparkQA · 2018-10-18T04:06:20Z

Test build #97514 has finished for PR 22675 at commit ba0d888.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-10-18T04:17:07Z

docs/ml-datasource.md

I would use SQL types consistently, for instance, StringType, IntegerType

HyukjinKwon · 2018-10-18T04:19:00Z

docs/ml-datasource.md

. -> ,.

HyukjinKwon · 2018-10-18T04:22:08Z

docs/ml-datasource.md

Should it be Datasource or Data sources? I am saying this because there looks a mismatch with the menu above.

Data sources.

HyukjinKwon · 2018-10-18T04:22:38Z

docs/ml-datasource.md

really personal preference tho .. like -> such as

HyukjinKwon · 2018-10-18T04:23:34Z

docs/ml-datasource.md

Shall we consistently make some codes such as StructType as codes like `StructType`

HyukjinKwon · 2018-10-18T04:24:20Z

Looks cool otherwise!

SparkQA · 2018-10-18T17:46:07Z

Test build #97541 has finished for PR 22675 at commit 1b54044.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2018-10-22T17:24:27Z

docs/ml-datasource.md

"some specific data sources for ML"

imatiach-msft · 2018-10-22T17:30:14Z

@WeichenXu123 this looks good - one recommendation though, in the examples it might be good to pass the param that ignores bad images so that the not-image.txt is not included in the output, it might confuse people who are new to the API - unless you feel strongly about having the not-image.txt output in the examples

imatiach-msft · 2018-10-22T17:24:59Z

docs/ml-datasource.md

is this convention, to have this text here in the table of contents? "* This will become a table of contents (this text will be scraped)."

Yes. This keep the same with other ML algo page.

ah, ok, great

imatiach-msft · 2018-10-22T17:25:47Z

docs/ml-datasource.md

minor: number of image channels (no "the")

SparkQA · 2018-10-23T03:21:57Z

Test build #97889 has finished for PR 22675 at commit 8231cb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2018-10-23T03:45:58Z

LGTM, this is great to have!

HyukjinKwon

Looks good from me as well

cloud-fan · 2018-10-25T15:03:36Z

thanks, merging to master/2.4!

## What changes were proposed in this pull request? Spark datasource for image/libsvm user guide ## How was this patch tested? Scala: <img width="1022" alt="1" src="https://user-images.githubusercontent.com/19235986/47330111-a4f2e900-d6a9-11e8-9a6f-609fb8cd0f8a.png"> Java: <img width="1019" alt="2" src="https://user-images.githubusercontent.com/19235986/47330114-a9b79d00-d6a9-11e8-97fe-c7e4b8dd5086.png"> Python: <img width="1022" alt="3" src="https://user-images.githubusercontent.com/19235986/47330120-afad7e00-d6a9-11e8-8a0c-4340c2af727b.png"> R: <img width="1024" alt="4" src="https://user-images.githubusercontent.com/19235986/47330126-b3410500-d6a9-11e8-9329-5e6217718edd.png"> Closes #22675 from WeichenXu123/add_image_source_doc. Authored-by: WeichenXu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 6540c2f) Signed-off-by: Wenchen Fan <[email protected]>

## What changes were proposed in this pull request? Spark datasource for image/libsvm user guide ## How was this patch tested? Scala: <img width="1022" alt="1" src="https://user-images.githubusercontent.com/19235986/47330111-a4f2e900-d6a9-11e8-9a6f-609fb8cd0f8a.png"> Java: <img width="1019" alt="2" src="https://user-images.githubusercontent.com/19235986/47330114-a9b79d00-d6a9-11e8-97fe-c7e4b8dd5086.png"> Python: <img width="1022" alt="3" src="https://user-images.githubusercontent.com/19235986/47330120-afad7e00-d6a9-11e8-8a0c-4340c2af727b.png"> R: <img width="1024" alt="4" src="https://user-images.githubusercontent.com/19235986/47330126-b3410500-d6a9-11e8-9329-5e6217718edd.png"> Closes apache#22675 from WeichenXu123/add_image_source_doc. Authored-by: WeichenXu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

HyukjinKwon reviewed Oct 9, 2018

View reviewed changes

srowen requested changes Oct 9, 2018

View reviewed changes

HyukjinKwon reviewed Oct 11, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Member

HyukjinKwon Oct 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do a simple transformation so that how the image datasource can be utilized?

HyukjinKwon reviewed Oct 18, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Member

HyukjinKwon Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use SQL types consistently, for instance, StringType, IntegerType

HyukjinKwon reviewed Oct 18, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Member

HyukjinKwon Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

. -> ,.

HyukjinKwon reviewed Oct 18, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Member

HyukjinKwon Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really personal preference tho .. like -> such as

HyukjinKwon reviewed Oct 18, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Member

HyukjinKwon Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we consistently make some codes such as StructType as codes like `StructType`

imatiach-msft reviewed Oct 22, 2018

View reviewed changes

docs/ml-datasource.md Outdated

Copy link

Contributor

imatiach-msft Oct 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"some specific data sources for ML"

imatiach-msft approved these changes Oct 22, 2018

View reviewed changes

WeichenXu123 added 6 commits October 23, 2018 09:35

init

fbb38cb

address comments

a7fe411

address comments

3866d53

address comments

72c63c8

address comments

6de0c28

address comments

8231cb2

WeichenXu123 force-pushed the add_image_source_doc branch from 1b54044 to 8231cb2 Compare October 23, 2018 01:59

HyukjinKwon approved these changes Oct 23, 2018

View reviewed changes

asfgit closed this in 6540c2f Oct 25, 2018

WeichenXu123 deleted the add_image_source_doc branch October 25, 2018 15:12

zhengruifeng mentioned this pull request Jul 29, 2019

[SPARK-25584][ML][DOC] datasource for libsvm user guide #25286

Closed

[SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide #22675

[SPARK-25347][ML][DOC] Spark datasource for image/libsvm user guide #22675

Uh oh!

Conversation

WeichenXu123 commented Oct 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Oct 9, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Oct 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Oct 9, 2018

Uh oh!

WeichenXu123 commented Oct 9, 2018

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 10, 2018

Uh oh!

SparkQA commented Oct 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Oct 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 18, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 18, 2018

Uh oh!

WeichenXu123 commented Oct 9, 2018 •

edited

Loading

HyukjinKwon Oct 9, 2018 •

edited

Loading

HyukjinKwon Oct 11, 2018 •

edited

Loading