Skip to content

[SPARK-11564][SQL][follow-up] clean up java tuple encoder #9567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-11564][SQL][follow-up] clean up java tuple encoder #9567

wants to merge 5 commits into from

Conversation

@cloud-fan
Copy link
Contributor

@cloud-fan cloud-fan commented Nov 9, 2015

We need to support custom classes like java beans and combine them into tuple, and it's very hard to do it with the TypeTag-based approach.
We should keep only the compose-based way to create tuple encoder.

This PR also move Encoder to org.apache.spark.sql

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 9, 2015

cc @marmbrus @rxin

@SparkQA
Copy link

SparkQA commented Nov 9, 2015

Test build #45362 has finished for PR 9567 at commit 7ed72ec.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 9, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Nov 9, 2015

Test build #45375 has finished for PR 9567 at commit 7ed72ec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

marmbrus
marmbrus reviewed Nov 9, 2015
View reviewed changes
sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java Outdated
Copy link
Contributor

@marmbrus marmbrus Nov 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this is very hard unless we expose the API for creating Tuples from encoders?

Copy link
Contributor Author

@cloud-fan cloud-fan Nov 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, I think so. We already have Expression.tuple and I think we can use that after some adjustment.

@marmbrus
Copy link
Contributor

marmbrus commented Nov 10, 2015

Can you also get rid of the unneeded methods typeTagOfTuple2 and the class encoders (or make them private)?

@SparkQA
Copy link

SparkQA commented Nov 10, 2015

Test build #45479 has finished for PR 9567 at commit 7427269.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 10, 2015

Test build #45482 has finished for PR 9567 at commit ae129d9.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 10, 2015

Test build #45490 has finished for PR 9567 at commit 2d9f885.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 10, 2015

Test build #45477 has finished for PR 9567 at commit 8f1e270.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 10, 2015

Test build #45493 has finished for PR 9567 at commit fb1fe8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Nov 10, 2015

Sorry, there are conflicts now.

cloud-fan
cloud-fan reviewed Nov 11, 2015
View reviewed changes
sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
Copy link
Contributor Author

@cloud-fan cloud-fan Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marmbrus renaming to Encoders works!

Copy link
Contributor

@marmbrus marmbrus Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh awesome!

@SparkQA
Copy link

SparkQA commented Nov 11, 2015

Test build #45588 has finished for PR 9567 at commit 9fc5456.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 11, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Nov 11, 2015

Test build #45624 has finished for PR 9567 at commit 9fc5456.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cloud-fan commented Nov 11, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 11, 2015

Test build #45631 has finished for PR 9567 at commit 9fc5456.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Nov 11, 2015

Thanks! Merging to master and 1.6.

asfgit pushed a commit that referenced this pull request Nov 11, 2015
@cloud-fan @marmbrus
[SPARK-11564][SQL][FOLLOW-UP] clean up java tuple encoder
9bf9885 
We need to support custom classes like java beans and combine them into tuple, and it's very hard to do it with the  TypeTag-based approach.
We should keep only the compose-based way to create tuple encoder.

This PR also move `Encoder` to `org.apache.spark.sql`

Author: Wenchen Fan <[email protected]>

Closes #9567 from cloud-fan/java.

(cherry picked from commit ec2b807)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in ec2b807 Nov 11, 2015
dskrvk pushed a commit to dskrvk/spark that referenced this pull request Nov 13, 2015
@cloud-fan @dskrvk
[SPARK-11564][SQL][FOLLOW-UP] clean up java tuple encoder
decbc73 
We need to support custom classes like java beans and combine them into tuple, and it's very hard to do it with the  TypeTag-based approach.
We should keep only the compose-based way to create tuple encoder.

This PR also move `Encoder` to `org.apache.spark.sql`

Author: Wenchen Fan <[email protected]>

Closes apache#9567 from cloud-fan/java.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cloud-fan @SparkQA @marmbrus