[SPARK-11500][SQL] Not deterministic order of columns when using merging schemas.

[HyukjinKwon]

https://issues.apache.org/jira/browse/SPARK-11500

As filed in SPARK-11500, if merging schemas is enabled, the order of files to touch is a matter which might affect the ordering of the output columns.

This was mostly because of the use of Set and Map so I replaced them to LinkedHashSet and LinkedHashMap to keep the insertion order.

Also, I changed reduceOption to reduceLeftOption, and replaced the order of filesToTouch from metadataStatuses ++ commonMetadataStatuses ++ needMerged to needMerged ++ metadataStatuses ++ commonMetadataStatuses in order to touch the part-files first which always have the schema in footers whereas the others might not exist.

One nit is, If merging schemas is not enabled, but when multiple files are given, there is no guarantee of the output order, since there might not be a summary file for the first file, which ends up putting ahead the columns of the other files.

However, I thought this should be okay since disabling merging schemas means (assumes) all the files have the same schemas.

In addition, in the test code for this, I only checked the names of fields.

[HyukjinKwon]

cc @liancheng

[cloud-fan]

ok to test

[yhuai]

add to whitelist

[SparkQA]

Test build #45235 has started for PR 9517 at commit bcf72d3.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #45324 has finished for PR 9517 at commit bcf72d3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

We should probably use a partitioned table here. Directories like base/table1, base/table2, and base/table3 are not valid partition directory names, and loading base as a Parquet file should throw an exception. It's not expected that this test case can pass.

[HyukjinKwon]

Thanks for commands!

[HyukjinKwon]

In this commit, I added partitioned tables for the test and sorted the FileStatuses.

There are several things to mention here.

Firstly, now we do not need to change Set to LinkedHashSet and Map to LinkedHashMap for this issue since it manually sorts the FileStatuses. However, I left them as I though anyway the order of files better be in the order as they are retrieved. If that looks weird, I would like to get it back.

Secondly, in any cases, the columns of the lexicographically first file head first, which might be a matter for files containing numeric values. However, I left this as I thought anyway it is deterministic.

[SparkQA]

Test build #45359 has finished for PR 9517 at commit 4f47063.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

I used sortBy instead of sortWith

[SparkQA]

Test build #45489 has finished for PR 9517 at commit 32dfb87.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

LGTM, merged to master. Thanks!

[liancheng]

Also backported to branch-1.6.