Skip to content

[SPARK-5821] [SQL] JSON CTAS command should throw error message when delete path failure #4610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

[SPARK-5821] [SQL] JSON CTAS command should throw error message when delete path failure #4610

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5a42d83
JSONRelation CTAS should check if delete is successful
Feb 16, 2015
e4bc229
Update JSONRelation.scala
yanboliang Mar 16, 2015
79f7040
Update JSONRelation.scala
yanboliang Mar 16, 2015
e2df8d5
check path exisit when write
yanboliang Mar 16, 2015
46f0d9d
Update JSONRelation.scala
yanboliang Mar 18, 2015
42d7fb6
add unittest & fix output format
yanboliang Mar 19, 2015
c387fce
fix typos
yanboliang Mar 19, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 29 additions & 7 deletions sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,23 @@ private[sql] class DefaultSource
mode match {
case SaveMode.Append =>
sys.error(s"Append mode is not supported by ${this.getClass.getCanonicalName}")
case SaveMode.Overwrite =>
fs.delete(filesystemPath, true)
case SaveMode.Overwrite => {
var success: Boolean = false
try {
success = fs.delete(filesystemPath, true)
} catch {
case e: IOException =>
throw new IOException(
s"Unable to clear output directory ${filesystemPath.toString} prior"
+ s" to writing to JSON table:\n${e.toString}")
}
if (!success) {
throw new IOException(
s"Unable to clear output directory ${filesystemPath.toString} prior"
+ s" to writing to JSON table.")
}
Copy link
Contributor

@yhuai yhuai Feb 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yanbohappy Seems we just throw another error message at here. Based on your JIRA description, I think you need to check if delete returns true or false when data already exists.

Copy link
Contributor Author

@yanboliang yanboliang Feb 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yhuai Yes, it is expected. If there is no exception thrown, it can ensure delete success. I have investigated other codes in spark and they do the same thing for delete.

true
}
case SaveMode.ErrorIfExists =>
sys.error(s"path $path already exists.")
case SaveMode.Ignore => false
Expand Down Expand Up @@ -109,13 +123,21 @@ private[sql] case class JSONRelation(
val fs = filesystemPath.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)

if (overwrite) {
try {
fs.delete(filesystemPath, true)
} catch {
case e: IOException =>
if (fs.exists(filesystemPath)) {
var success: Boolean = false
try {
success = fs.delete(filesystemPath, true)
} catch {
case e: IOException =>
throw new IOException(
s"Unable to clear output directory ${filesystemPath.toString} prior"
+ s" to writing to JSON table:\n${e.toString}")
}
if (!success) {
throw new IOException(
s"Unable to clear output directory ${filesystemPath.toString} prior"
+ s" to INSERT OVERWRITE a JSON table:\n${e.toString}")
+ s" to writing to JSON table.")
}
}
// Write the data.
data.toJSON.saveAsTextFile(path)
Expand Down
25 changes: 24 additions & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

package org.apache.spark.sql.sources

import java.io.File
import java.io.{IOException, File}

import org.scalatest.BeforeAndAfterAll

Expand Down Expand Up @@ -62,6 +62,29 @@ class CreateTableAsSelectSuite extends DataSourceTest with BeforeAndAfterAll {
dropTempTable("jsonTable")
}

test("CREATE TEMPORARY TABLE AS SELECT based on the file without write permission") {
val childPath = new File(path.toString, "child")
path.mkdir()
childPath.createNewFile()
path.setWritable(false)

val e = intercept[IOException] {
sql(
s"""
|CREATE TEMPORARY TABLE jsonTable
|USING org.apache.spark.sql.json.DefaultSource
|OPTIONS (
| path '${path.toString}'
|) AS
|SELECT a, b FROM jt
""".stripMargin)
sql("SELECT a, b FROM jsonTable").collect()
}
assert(e.getMessage().contains("Unable to clear output directory"))

path.setWritable(true)
}

test("create a table, drop it and create another one with the same name") {
sql(
s"""
Expand Down