[SPARK-17326][SPARKR] Fix tests with HiveContext in SparkR not to be skipped always

[HyukjinKwon]

What changes were proposed in this pull request?

Currently, HiveContext in SparkR is not being tested and always skipped.
This is because the initiation of TestHiveContext is being failed due to trying to load non-existing data paths (test tables).

This is introduced from #14005

This enables the tests with SparkR.

How was this patch tested?

Manually,

Before (on Mac OS)

...
Skipped ------------------------------------------------------------------------
1. create DataFrame from RDD (@test_sparkSQL.R#200) - Hive is not build with SparkSQL, skipped
2. test HiveContext (@test_sparkSQL.R#1041) - Hive is not build with SparkSQL, skipped
3. read/write ORC files (@test_sparkSQL.R#1748) - Hive is not build with SparkSQL, skipped
4. enableHiveSupport on SparkSession (@test_sparkSQL.R#2480) - Hive is not build with SparkSQL, skipped
5. sparkJars tag in SparkContext (@test_Windows.R#21) - This test is only for Windows, skipped
...

After (on Mac OS)

...
Skipped ------------------------------------------------------------------------
1. sparkJars tag in SparkContext (@test_Windows.R#21) - This test is only for Windows, skipped
...

Please refer the tests below (on Windows)

• Before: https://ci.appveyor.com/project/HyukjinKwon/spark/build/45-test123
• After: https://ci.appveyor.com/project/HyukjinKwon/spark/build/46-test123

[HyukjinKwon]

cc @rxin, @felixcheung and @shivaram

[shivaram]

Thanks @HyukjinKwon - This is a great catch. LGTM pending tests.

[felixcheung]

LGTM thanks!

[SparkQA]

Test build #64700 has finished for PR 14889 at commit 2cdcf4f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

btw, do you know why the "skipped" message is not in the Jenkins log?

[HyukjinKwon]

@felixcheung I remember in case of PySpark/Scala, only fixed and related modules are tested by tracking the changed/related modules via changed files in the PR with Jenkins (within run-tests.py) but not too sure of SparkR one.

Ah, you meant just message within Skipped -------------------------------------------------------------------- block. I see. I will check this out.

[HyukjinKwon]

It seems SparkR always runs the tests all? I will check this out.

[felixcheung]

thanks for fixing this. I thought skipped tests are easy to miss so it would be great (maybe as a separate PR) if there is a way we could detect what's being skipped from within Jenkins.

[HyukjinKwon]

@felixcheung It seems due to the differences of testthat version..? It seems this printing is added from testthat 1.0.0(https://github.com/hadley/testthat/blob/master/NEWS.md#testthat-100) (with this r-lib/testthat@57dc3ce)?

Do you mind if I ask what version is used in Jenkins (if you know)? It seems I can't find.

[felixcheung]

shane knapp manages Jenkins, he should know. Does the newer testthat have any minimal R version requirement? We are running R 3.1.1 (the minimal required R version) in Jenkins.

[HyukjinKwon]

It seems okay with minimal version. It seems 3.1+ (if my understanding is correct, https://github.com/hadley/testthat/blob/v1.0.0/DESCRIPTION#L13). I will try to look into this deeper.

[shivaram]

I just ran this on one of the Jenkins machines

> packageVersion("testthat")
[1] ‘0.11.0’

[felixcheung]

could we update testthat on Jenkins box? I think that would automagically bubble up the skipped test.
maybe then we could parse the log to check if we are not accidentally skipping more test over time?

[felixcheung]

let's merge this fix and follow up if there is a better way to track skipped tests separately?

[shivaram]

Yeah lets open a separate JIRA to update testthat on the Jenkins boxes. LGTM. Merging this to master and branch-2.0