Skip to content

[SPARK-29200][SQL] Optimize extract/date_part for epoch #25881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-29200][SQL] Optimize extract/date_part for epoch #25881

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1dc1360
Optimize getEpoch
MaxGekk Sep 21, 2019
7e90f72
Regenerate benchmark results
MaxGekk Sep 21, 2019
380fa21
Merge remote-tracking branch 'remotes/origin/master' into optimize-ex…
MaxGekk Sep 22, 2019
bdedf26
Regen benchmark results
MaxGekk Sep 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,9 +847,10 @@ object DateTimeUtils {
* since 1970-01-01 00:00:00 local time.
*/
def getEpoch(timestamp: SQLTimestamp, zoneId: ZoneId): Decimal = {
val offset = zoneId.getRules.getOffset(microsToInstant(timestamp)).getTotalSeconds
val sinceEpoch = BigDecimal(timestamp) / MICROS_PER_SECOND + offset
new Decimal().set(sinceEpoch, 20, 6)
val offset = SECONDS.toMicros(
zoneId.getRules.getOffset(microsToInstant(timestamp)).getTotalSeconds)
val sinceEpoch = timestamp + offset
Decimal(sinceEpoch, 20, 6)
}

def currentTimestamp(): SQLTimestamp = instantToMicros(Instant.now())
Expand Down
Loading