Skip to content

Updated the Spark SQL Programming guide with Custom object encoding for Dataset and unsupported operation error handling #16997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Updated the Spark SQL Programming guide with Custom object encoding for Dataset and unsupported operation error handling #16997

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
103906f
Updated the SQL programming guide to explain about the Encoding opera…
HarshSharma8 Feb 20, 2017
9c8f63c
Updated the docs to match the voice of my updates in SQL Programming …
HarshSharma8 Feb 20, 2017
7a539a7
Modified the content and replaced the code block inside back-ticks in…
HarshSharma8 Feb 21, 2017
d49ae50
Updated the content to provide object name
HarshSharma8 Feb 21, 2017
c2fd0ad
Updated the content to provide object name
HarshSharma8 Feb 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,9 @@ reflection and become the names of the columns. Case classes can also be nested
types such as `Seq`s or `Array`s. This RDD can be implicitly converted to a DataFrame and then be
registered as a table. Tables can be used in subsequent SQL statements.

Spark Encoders are used to convert a JVM object to Spark SQL representation. To create a Dataset, Spark requires an encoder which takes the form of `Encoder[T]` where `T` is the type which has to be encoded.
Considering an object of class `DemoObj(id: Int, name: String)` as a type of the Dataset to be created, may result into java.lang.UnsupportedOperationException: No Encoder found for DemoObj. To overcome this problem, the kryo encoder is used. It generally tells Spark SQL to encode object of `DemoObj`, so that the operation could find the encoded `DemoObj`.

{% include_example schema_inferring scala/org/apache/spark/examples/sql/SparkSQLExample.scala %}
</div>

Expand Down