Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Sep 21, 2019

What changes were proposed in this pull request?

Refactoring of the DateTimeUtils.getEpoch() function by avoiding decimal operations that are pretty expensive, and converting the final result to the decimal type at the end.

Why are the changes needed?

The changes improve performance of the getEpoch() method at least up to 20 times.
Before:

Invoke extract for timestamp:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
cast to timestamp                                   256            277          33         39.0          25.6       1.0X
EPOCH of timestamp                                23455          23550         131          0.4        2345.5       0.0X

After:

Invoke extract for timestamp:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
cast to timestamp                                   255            294          34         39.2          25.5       1.0X
EPOCH of timestamp                                 1049           1054           9          9.5         104.9       0.2X

Does this PR introduce any user-facing change?

No

How was this patch tested?

By existing test from DateExpressionSuite.

@SparkQA
Copy link

SparkQA commented Sep 21, 2019

Test build #111115 has finished for PR 25881 at commit 7e90f72.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

…tract-epoch

# Conflicts:
#	sql/core/benchmarks/ExtractBenchmark-results.txt
@SparkQA
Copy link

SparkQA commented Sep 22, 2019

Test build #111143 has finished for PR 25881 at commit bdedf26.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

@MaxGekk MaxGekk deleted the optimize-extract-epoch branch October 5, 2019 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants