[SPARK-6696] [SQL] Adds HiveContext.refreshTable to PySpark

[liancheng]

[SparkQA]

Test build #29671 has started for PR 5349 at commit 004bec0.

[SparkQA]

Test build #29671 has finished for PR 5349 at commit 004bec0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29671/
Test PASSed.

[liancheng]

Didn't write a test for this because we don't have Hive testing infra like TestHive in Scala code for PySpark yet. I opened this PR to make code snippets in #5348 more complete. But then I realize we can always use sqlCtx.sql("REFRESH TABLE my_table") in Python. So this PR doesn't seem that necessary.

Tested this locally with the following snippet:

import tempfile, shutil

jsonFile = tempfile.mkdtemp()
shutil.rmtree(jsonFile)
sqlCtx.createDataFrame([("a", "b")])\
      .toJSON()\
      .saveAsTextFile(jsonFile)
sqlCtx.sql(
    "CREATE TABLE jt " +
    "USING org.apache.spark.sql.json " +
    "OPTIONS (path '%s')" % jsonFile)
sqlCtx.sql("SELECT * FROM jt").show()
# _1 _2
# a  b

shutil.rmtree(jsonFile)
sqlCtx.createDataFrame([("a", "b", "c")])\
      .toJSON()\
      .saveAsTextFile(jsonFile)
sqlCtx.sql("SELECT * FROM jt").show()
# _1 _2
# a  b

sqlCtx.refreshTable("jt")
sqlCtx.sql("SELECT * FROM jt").show()
# _1 _2 _3
# a  b  c

[rxin]

Can't we add that testing snippet to tests.py?

[liancheng]

We need a properly configured Hive environment to run the test. I can add a simple TestHive-like class to do metastore / warehouse configurations though.

[marmbrus]

Thanks, I merged this to master. Would be great to add a TestHive wrapper in a followup PR.