Skip to content

[SPARK-29074][SQL] Optimize date_format for foldable fmt #25782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

[SPARK-29074][SQL] Optimize date_format for foldable fmt #25782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
de8dafb
Optimize date_format for foldable fmt
MaxGekk Sep 13, 2019
5ab324f
Re-gen results of DateTimeBenchmark
MaxGekk Sep 13, 2019
85a290e
Bug fix: copy formatter
MaxGekk Sep 13, 2019
ee2a7d2
Re-gen results of DateTimeBenchmark
MaxGekk Sep 13, 2019
3292acc
Use default locale
MaxGekk Sep 14, 2019
3b5a8c2
Skip DateTimeUtils
MaxGekk Sep 14, 2019
0242852
Use @transient private lazy val formatter
MaxGekk Sep 16, 2019
a9fd8e4
Handle null from right.eval()
MaxGekk Sep 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions ...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -592,19 +592,35 @@ case class DateFormatClass(left: Expression, right: Expression, timeZoneId: Opti
override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression =
copy(timeZoneId = Option(timeZoneId))

@transient private lazy val formatter: Option[TimestampFormatter] = {
if (right.foldable) {
Option(right.eval()).map(format => TimestampFormatter(format.toString, zoneId))
} else None
}

override protected def nullSafeEval(timestamp: Any, format: Any): Any = {
val df = TimestampFormatter(format.toString, zoneId)
UTF8String.fromString(df.format(timestamp.asInstanceOf[Long]))
val tf = if (formatter.isEmpty) {
Copy link
Member

@srowen srowen Sep 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about .getOrElse?

Copy link
Member Author

@MaxGekk MaxGekk Sep 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.getOrElse has some overhead of calling the lambda function. I explicitly avoided its usage in the interpreted mode. For consistency, I could do the same in the codegen function but I don't think it does matter.

TimestampFormatter(format.toString, zoneId)
} else {
formatter.get
}
UTF8String.fromString(tf.format(timestamp.asInstanceOf[Long]))
}

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
val locale = ctx.addReferenceObj("locale", Locale.US)
defineCodeGen(ctx, ev, (timestamp, format) => {
s"""UTF8String.fromString($tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
formatter.map { tf =>
val timestampFormatter = ctx.addReferenceObj("timestampFormatter", tf)
defineCodeGen(ctx, ev, (timestamp, _) => {
s"""UTF8String.fromString($timestampFormatter.format($timestamp))"""
})
}.getOrElse {
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
defineCodeGen(ctx, ev, (timestamp, format) => {
s"""UTF8String.fromString($tf$$.MODULE$$.apply($format.toString(), $zid)
.format($timestamp))"""
})
})
}
}

override def prettyName: String = "date_format"
Expand Down
4 changes: 2 additions & 2 deletions sql/core/benchmarks/DateTimeBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,8 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
format date: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
format date wholestage off 7180 / 7181 1.4 718.0 1.0X
format date wholestage on 7051 / 7194 1.4 705.1 1.0X
format date wholestage off 4787 / 4839 2.1 478.7 1.0X
format date wholestage on 4736 / 4802 2.1 473.6 1.0X


================================================================================================
Expand Down