-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-18714][SQL] Add a simple time function to SparkSession #16140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
cc @marmbrus |
|
Test build #69662 has started for PR 16140 at commit |
|
Retest this please |
|
It's ok no need to retest. The change is fine. |
|
I see. BTW, @rxin . Do we need |
|
That's much more difficult to type though. |
|
True. Never mind. I just thought we were able to use like the following. |
|
Test build #69664 has finished for PR 16140 at commit
|
|
shrug It seems pretty unrelated to SparkSession. I get that it's a convenience method but something Spark really needs to expose as an API method? |
|
The cost to maintain this seems very small though, and I'd definitely use it all the time in the repl. In Databricks this is not an issue since the environment always appends the time, but I really miss this in the repl. |
|
LGTM - merging to master/2.1. Thanks! |
## What changes were proposed in this pull request?
Many Spark developers often want to test the runtime of some function in interactive debugging and testing. This patch adds a simple time function to SparkSession:
```
scala> spark.time { spark.range(1000).count() }
Time taken: 77 ms
res1: Long = 1000
```
## How was this patch tested?
I tested this interactively in spark-shell.
Author: Reynold Xin <[email protected]>
Closes #16140 from rxin/SPARK-18714.
(cherry picked from commit cb1f10b)
Signed-off-by: Herman van Hovell <[email protected]>
## What changes were proposed in this pull request?
Many Spark developers often want to test the runtime of some function in interactive debugging and testing. This patch adds a simple time function to SparkSession:
```
scala> spark.time { spark.range(1000).count() }
Time taken: 77 ms
res1: Long = 1000
```
## How was this patch tested?
I tested this interactively in spark-shell.
Author: Reynold Xin <[email protected]>
Closes apache#16140 from rxin/SPARK-18714.
## What changes were proposed in this pull request?
Many Spark developers often want to test the runtime of some function in interactive debugging and testing. This patch adds a simple time function to SparkSession:
```
scala> spark.time { spark.range(1000).count() }
Time taken: 77 ms
res1: Long = 1000
```
## How was this patch tested?
I tested this interactively in spark-shell.
Author: Reynold Xin <[email protected]>
Closes apache#16140 from rxin/SPARK-18714.
What changes were proposed in this pull request?
Many Spark developers often want to test the runtime of some function in interactive debugging and testing. This patch adds a simple time function to SparkSession:
How was this patch tested?
I tested this interactively in spark-shell.