[SPARK-12562][SQL] DataFrame.write.format(text) requires the column name to be called value

[xguo27]

No description provided.

[xguo27]

@marmbrus Thanks Michael for your feedback!

Looks like the 'value' is to give the single string column a arbitrary name. Current implementation strips schema information when creating TextRelation (after verifying the schema is single field with string type). It is fine during read, but fails during write.

Would you mind taking another look at my updated change?

[viirya]

The comment should be updated too.

[viirya]

Can you add some tests for this?

[xguo27]

Thanks @viirya ! I have updated the comment and added unit test.

[rxin]

why don't you just change the existing test case to rename the dataframe column and leave the following as a comment there?

SPARK-12562 verify write.text() can handle column name beyond value

[xguo27]

@rxin I thought about it, but was not sure if it was a good idea to change the existing testcase. In the existing test, should I add a second dataframe with column renamed, or just replace the original dataframe with column renaming?

[rxin]

Just replace the original one to something weird, like "adwrasdf"

[viirya]

This will be failed because the later method verifyFrame will check if the df read in has schema like new StructType().add("value", StringType). You could update verifyFrame to check if it has only one StringType column.

[xguo27]

After write.text(), the local text file actually does not carry the schema name like JSON does. When reading back the text file and then call verifyFrame, it will always have value as the column name.

[viirya]

yes. it is right.

[xguo27]

@marmbrus Can we trigger a test for this?

[SparkQA]

Test build #2309 has finished for PR 10515 at commit d01ecac.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - I've merged this.

[cloud-fan]

should we make sure that textSchema is a struct type that has only one string field?

[xguo27]

@cloud-fan DefaultSource.scala is the only place that creates a TextRelation, and it verifies that the schema is size 1 and of type string before creating a TextRelation. So I think it is fine not to verify again here. What do you think?

[cloud-fan]

oh, then it's fine