Skip to content

Commit 48b459d

Browse files
JoshRosenrxin
authored andcommitted
[SPARK-17301][SQL] Remove unused classTag field from AtomicType base class
There's an unused `classTag` val in the AtomicType base class which is causing unnecessary slowness in deserialization because it needs to grab ScalaReflectionLock and create a new runtime reflection mirror. Removing this unused code gives a small but measurable performance boost in SQL task deserialization. Author: Josh Rosen <[email protected]> Closes #14869 from JoshRosen/remove-unused-classtag.
1 parent 736a791 commit 48b459d

File tree

1 file changed

+1
-9
lines changed

1 file changed

+1
-9
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,10 @@
1717

1818
package org.apache.spark.sql.types
1919

20-
import scala.reflect.ClassTag
21-
import scala.reflect.runtime.universe.{runtimeMirror, TypeTag}
20+
import scala.reflect.runtime.universe.TypeTag
2221

2322
import org.apache.spark.annotation.DeveloperApi
24-
import org.apache.spark.sql.catalyst.ScalaReflectionLock
2523
import org.apache.spark.sql.catalyst.expressions.Expression
26-
import org.apache.spark.util.Utils
2724

2825
/**
2926
* A non-concrete data type, reserved for internal uses.
@@ -130,11 +127,6 @@ protected[sql] abstract class AtomicType extends DataType {
130127
private[sql] type InternalType
131128
private[sql] val tag: TypeTag[InternalType]
132129
private[sql] val ordering: Ordering[InternalType]
133-
134-
@transient private[sql] val classTag = ScalaReflectionLock.synchronized {
135-
val mirror = runtimeMirror(Utils.getSparkClassLoader)
136-
ClassTag[InternalType](mirror.runtimeClass(tag.tpe))
137-
}
138130
}
139131

140132

0 commit comments

Comments
 (0)