[SPARK-22926] [SQL] Respect table-level conf compression codec Compression in multiple scenarios

[gatorsmile]

What changes were proposed in this pull request?

Currently, we ignore table-specific compression conf when the Hive serde tables are converted to the data source tables. We also ignore it when users set compression in the TBLPROPERTIES clause instead of the OPTIONS clause, even if the tables are native data source tables. This PR is to fix the above issue.

The PR #20087 will make the Hive-serde writer aware of the table-level prop.

The PR #20076 will make Spark aware of the table-specific conf parquet.compression too

How was this patch tested?

Added test cases

[gatorsmile]

cc @cloud-fan

[SparkQA]

Test build #85538 has finished for PR 20120 at commit 7189562.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

retest this please

[SparkQA]

Test build #85546 has finished for PR 20120 at commit 7189562.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I think we should document the difference between table options and properties. AFAIK we added table properties to data source tables since Spark 2.3, and previously table options is the only place for users to put some configs to change some behaviors.

[dongjoon-hyun]

@gatorsmile . Did you add some document for this?