[SPARK-19570][PYSPARK] Allow to disable hive in pyspark shell #16906
Conversation
|
Test build #72796 has finished for PR 16906 at commit
|
|
@holdenk Please help review |
|
Let me take a look tomorrow. |
|
Ah interesting, thanks for working on this. I've been working on a JIRA which also requires "stand alone" style testing, I'll make the (WIP) PR for that and maybe you can take a look at it. Perhaps we could generalize that one a bit and make something for testing things like this (provided the overhead isn't too high). |
|
I think we need to lowercase the conf value as in #16907 |
|
Yeah, make sense. Fixed it. |
|
Test build #74944 has finished for PR 16906 at commit
|
|
Test build #75428 has finished for PR 16906 at commit
|
felixcheung
left a comment
felixcheung
left a comment
There was a problem hiding this comment.
LGTM, @holdenk do you have any comment on this?
|
I think this looks reasonable, although it would maybe make sense to add a warning if the user has explicitly requested hive support and we are falling through to non-hive support (e.g. in the except side of the try block). |
|
+1 on that, we do have the log on the R side. |
|
Test build #75562 has finished for PR 16906 at commit
|
|
Thanks @holdenk @felixcheung for review, I just added the warning log |
|
Kindly ping @holdenk |
|
LGTM, thanks for working on this @zjffdu |
|
Merged to master |
## What changes were proposed in this pull request? SPARK-15236 do this for scala shell, this ticket is for pyspark shell. This is not only for pyspark itself, but can also benefit downstream project like livy which use shell.py for its interactive session. For now, livy has no control of whether enable hive or not. ## How was this patch tested? I didn't find a way to add test for it. Just manually test it. Run `bin/pyspark --master local --conf spark.sql.catalogImplementation=in-memory` and verify hive is not enabled. Author: Jeff Zhang <[email protected]> Closes apache#16906 from zjffdu/SPARK-19570.
4 participants
What changes were proposed in this pull request?
SPARK-15236 do this for scala shell, this ticket is for pyspark shell. This is not only for pyspark itself, but can also benefit downstream project like livy which use shell.py for its interactive session. For now, livy has no control of whether enable hive or not.
How was this patch tested?
I didn't find a way to add test for it. Just manually test it.
Run
bin/pyspark --master local --conf spark.sql.catalogImplementation=in-memoryand verify hive is not enabled.