[SPARK-27552][SQL]The configuration hive.exec.stagingdir is invalid on Windows OS

[10110346]

What changes were proposed in this pull requesst?

If we set hive.exec.stagingdir=.test-staging\tmp,
But the staging directory is still .hive-staging on Windows OS.

Reasons for failure:
Test code:

 val path = new Path("C:\\test\\hivetable")
  println("path.toString: " + path.toString)
  println("path.toUri.getPath: " + path.toUri.getPath)

Output:

path.toString: C:/test/hivetable
path.toUri.getPath: /C:/test/hivetable

We can see that path.toUri.getPath has one more separator than path.toString, and the separator is ' / ', not ' \ '
So stagingPathName.stripPrefix(inputPathName).stripPrefix(File.separator).startsWith(".") will return false

How was this patch tested?

1. Existed tests
2. Manual testing on Windows OS

[SparkQA]

Test build #104849 has finished for PR 24446 at commit 8544174.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

This drops theme:

scala> new Path("hdfs://0.0.0:8020/a/b/c").toString
res1: String = hdfs://0.0.0:8020/a/b/c

scala> new Path("hdfs://0.0.0:8020/a/b/c").toUri.getPath
res2: String = /a/b/c

So, if the theme is different from the default, it looks it's going to be ignored.

[dongjoon-hyun]

+1 for @HyukjinKwon 's comment.

[10110346]

Can we change this codes:

val inputPathUri: URI = inputPath.toUri
val inputPathName: String = inputPathUri.getPath

to:
val inputPathName: String = inputPath.toString

[HyukjinKwon]

Can you add a set of tests for this method getStagingDir?

[gaborgsomogyi]

+1 not to use toUri, please see the following:

scala> import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.Path

scala> val f = new java.io.File("/tmp/chk %#chk")
f: java.io.File = /tmp/chk %#chk

scala> new Path(new Path(f.toURI()), "foo").toString
res0: String = file:/tmp/chk %#chk/foo

scala> new Path(new Path(f.toURI()), "foo").toUri.toString
res1: String = file:/tmp/chk%20%25%23chk/foo

[HyukjinKwon]

Yea, it's a common mistake for double-encoded problem.

[10110346]

add a set of tests

Sorry for the late reply.
I'll try to add some unit tests.

[gaborgsomogyi]

invokePrivate?

[10110346]

Many members are undefined in SaveAsHiveFile trait, so object creation impossible for it .

[gaborgsomogyi]

Then why not just private[hive] def getStagingDir?

[10110346]

OK, it looks better, I will change it, thanks

[SparkQA]

Test build #105247 has finished for PR 24446 at commit b8aad31.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gaborgsomogyi]

Looks like relevant issue:

java.lang.IllegalStateException: Cannot create staging directory  'file:/a/b_hive_2019-05-08_01-34-08_839_7862489166644050807-1'

[SparkQA]

Test build #105251 has finished for PR 24446 at commit e34ecae.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #105254 has finished for PR 24446 at commit c02e8ba.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gaborgsomogyi]

LGTM.

[srowen]

@HyukjinKwon does this address your comment?

[srowen]

Merged to master