Skip to content

[SPARK-13540][SQL] Supports using nested classes within Scala objects as Dataset element type #11421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
+19 −1
Closed

[SPARK-13540][SQL] Supports using nested classes within Scala objects as Dataset element type #11421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc225af
Supports Scala object as outer scope
liancheng Feb 28, 2016
d8ca6d7
Fixes the problem in the right way
liancheng Feb 28, 2016
e707881
Addresses PR comments
liancheng Feb 29, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

package org.apache.spark.sql.catalyst.analysis

import java.lang.reflect.Modifier

import scala.collection.mutable.ArrayBuffer

import org.apache.spark.sql.AnalysisException
Expand Down Expand Up @@ -559,7 +561,13 @@ class Analyzer(
}

resolveExpression(unbound, LocalRelation(attributes), throws = true) transform {
case n: NewInstance if n.outerPointer.isEmpty && n.cls.isMemberClass =>
case n: NewInstance
Copy link
Contributor

@rxin rxin Feb 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be great to add some inline documentation what about these conditions are checking

// If this is an inner class of another class, register the outer object in `OuterScopes`.
// Note that static inner classes (e.g., inner classes within Scala objects) don't need
// outer pointer registration.
if n.outerPointer.isEmpty &&
n.cls.isMemberClass &&
!Modifier.isStatic(n.cls.getModifiers) =>
val outer = OuterScopes.outerScopes.get(n.cls.getDeclaringClass.getName)
if (outer == null) {
throw new AnalysisException(
Expand Down
10 changes: 10 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,12 +621,22 @@ class DatasetSuite extends QueryTest with SharedSQLContext {
ds.filter(_ => true),
Some(1), Some(2), Some(3))
}

test("SPARK-13540 Dataset of nested class defined in Scala object") {
checkAnswer(
Seq(OuterObject.InnerClass("foo")).toDS(),
OuterObject.InnerClass("foo"))
}
}

class OuterClass extends Serializable {
case class InnerClass(a: String)
}

object OuterObject {
case class InnerClass(a: String)
}

case class ClassData(a: String, b: Int)
case class ClassData2(c: String, d: Int)
case class ClassNullableData(a: String, b: Integer)
Expand Down