SPARK-18086: Add support for Hive session vars.

[rdblue]

What changes were proposed in this pull request?

This adds support for Hive variables:

• Makes values set via spark-sql --hivevar name=value accessible
• Adds getHiveVar and setHiveVar to the HiveClient interface
• Adds a SessionVariables trait for sessions like Hive that support variables (including Hive vars)
• Adds SessionVariables support to variable substitution
• Adds SessionVariables support to the SET command

How was this patch tested?

• Adds a test to all supported Hive versions for accessing Hive variables
• Adds HiveVariableSubstitutionSuite

[rxin]

How are the session vars different from our configs? Can't we just use that?

[rdblue]

The purpose of this is to ensure the same behavior as Hive so users can move to SparkSQL without rewriting queries. That's why I'm trying to keep behavior as close as possible to the behavior of Hive (and Spark 1.6.1).

Session vars are kept in a different namespace, so collisions should not overwrite configs and session vars take precedence in Hive (actually, conf substitution requires the "hiveconf:" prefix, otherwise values are only taken from session vars). Because we still use a Hive SessionState and pass some commands to Hive, I thought it was also important to store the hive variables in the session state like Hive would. That way any command that are passed to Hive that have variable references work as well.

[SparkQA]

Test build #68004 has finished for PR 15738 at commit 8c1b7b4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SparkSqlParser(
  • trait SessionVariables
  • class VariableSubstitution(conf: SQLConf, sessionVariables: Option[SessionVariables] = None)

[SparkQA]

Test build #68301 has finished for PR 15738 at commit 45a4cee.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #68303 has finished for PR 15738 at commit 78a95fa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging in master/branch-2.1!

[rdblue]

Thanks for reviewing @rxin!