[SPARK-6941][SQL] Provide a better error message to when inserting into RDD based table

[yjshen]

JIRA: https://issues.apache.org/jira/browse/SPARK-6941

[yjshen]

@yhuai it seems not possible to do checking in CheckAnalysis because LogicalRelation or LogicalRDD is not accessible there?

[yhuai]

You can put it in org.apache.spark.sql.sources.PreWriteCheck (it is a extendedCheckRules configured in SQLContext and HiveContext), which is located in sql/core. Eventually, we will move it out from the package of sources since this is another round of write related checks.

[yjshen]

thanks, I would update my implementation :)

[rxin]

"Inserting into an RDD-based table is not allowed."

[yhuai]

ok to test

[SparkQA]

Test build #37119 has finished for PR 7342 at commit 68324a1.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1052 has finished for PR 7342 at commit 68324a1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37225 has finished for PR 7342 at commit 0c635d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

fix import order here

[SparkQA]

Test build #37303 has finished for PR 7342 at commit 09518af.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Jenkins, retest this please.

[yhuai]

For this test suite, can you change class DataFrameSuite extends QueryTest { to class DataFrameSuite extends QueryTest with SQLTestUtils {? Then, you can use withTempPath to create a temp dir and save data in this temp dir. withTempPath will automatically clean up all of your temp data/dirs. Basically, you can do

test("SPARK-6941: Better error message for inserting into RDD-based Table") {
  withTempPath { dir =>
    // You can create some dirs inside the given temp dir.
    val tempParquet = new File(dir, "tmp_parquet")
    val tempJson = new File(dir, "tmp_json")
    ...
    // Your test code, which uses created temp dirs.
    ...
  }
}

[yhuai]

Looks good. Left a few comments about the test.

[SparkQA]

Test build #37394 has finished for PR 7342 at commit 09518af.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Thanks for the update! LGTM. I will merge it to master once jenkins passes.

[SparkQA]

Test build #37513 has finished for PR 7342 at commit 3d3f5dc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37516 has finished for PR 7342 at commit f82cbe7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yjshen]

@yhuai, Thanks for the step by step guide, really appreciate it.