SPARK-1349: spark-shell gets its own command history #267
Conversation
…ala repl This fix is simply a modification of the default FileBackedHistory file setting: https://github.com/scala/scala/blob/master/src/repl/scala/tools/nsc/interpreter/session/FileBackedHistory.scala#L77
|
Merged build triggered. Build is starting -or- tests failed to complete. |
|
Merged build started. Build is starting -or- tests failed to complete. |
|
Merged build finished. Build is starting -or- tests failed to complete. |
|
Build is starting -or- tests failed to complete. |
|
Jenkins, retest this please. |
|
Merged build triggered. Build is starting -or- tests failed to complete. |
|
Merged build started. Build is starting -or- tests failed to complete. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
There was a problem hiding this comment.
Just wondering - does this handle systems with different path separators correctly?
Does historyFile need to be scala.reflect.io.File? Or can it be a java.io.File? It seems a little heavyweight to pull in the reflection library to create a file path, but it makes sense if this is the needed type of historyFile.
There was a problem hiding this comment.
The reason I did this was to fully mimic the code from the Scala interpreter. I think this will make it easier to compare in case the Scala version is updated some time in the future, but you're right we could use a lighter weight construct, we just risk having different behavior than the standard Scala shell history.
Not a big deal to me, but I'm inclined to stick with cloning the Scala shell behavior. Let me know if you still prefer the java.io.File route, I'd be happy to change to that.
There was a problem hiding this comment.
Okay sounds good - just wondering why it was there.
|
Thanks, merged. |
Currently, spark-shell shares its command history with scala repl. This fix is simply a modification of the default FileBackedHistory file setting: https://github.com/scala/scala/blob/master/src/repl/scala/tools/nsc/interpreter/session/FileBackedHistory.scala#L77 Author: Aaron Davidson <[email protected]> Closes apache#267 from aarondav/repl and squashes the following commits: f9c62d2 [Aaron Davidson] SPARK-1349: spark-shell gets its own command history separate from scala repl
…oud and orange clouds (apache#267) * Add support for running ansible fuctional tests against opentelekomcloud and orange clouds
…33] Backport insert operation lock (apache#197) * [HADP-40184]Backport insert operation lock (#15) [HADP-31946] Fix data duplicate on application retry and support concurrent write to different partitions in the same table.[HADP-33040][HADP-33041] Optimize merging staging files to output path and detect conflict with HDFS file lease. HADP-34738] During commitJob, merge paths with multi threads (apache#218) [HADP-36251] Enhance the concurrent lock mechanism for insert operation (apache#272) [HADP-37137] Add option to disable insert operation lock to write partitioned table (apache#286) * [HADP-46224] Do not overwrite the lock file when creating lock (apache#133) * [HADP-46868] Fix Spark merge path race condition (apache#161) * [HADP-50903] Ignore the error message if insert operation lock file has been deleted (apache#271) * [HADP-50733] Enhance the error message on picking insert operation lock failure (apache#267) * Fix * Fix * Fix * fix * Fix * Fix * Fix * Fix * Fix * [HADP-50574] Support to create the lock file for EC enabled path (apache#263) * [HADP-50574][FOLLOWUP] Add parameter type when getting overwrite method (apache#265) * [HADP-50574][FOLLOWUP] Add UT for creating ec disabled lock file and use underlying DistributedFileSystem for ViewFileSystem (apache#266) * Fix * Fix * Fix * [HADP-34612][FOLLOWUP] Do not show the insert local error by removing the being written stream from dfs client (apache#288) * Enabled Hadoop 3 --------- Co-authored-by: fwang12 <[email protected]>
3 participants
Currently, spark-shell shares its command history with scala repl.
This fix is simply a modification of the default FileBackedHistory file setting:
https://github.com/scala/scala/blob/master/src/repl/scala/tools/nsc/interpreter/session/FileBackedHistory.scala#L77