[SPARK-16904] [SQL] Removal of Hive Built-in Hash Functions and TestHiveFunctionRegistry

[gatorsmile]

What changes were proposed in this pull request?

Currently, the Hive built-in hash function is not being used in Spark since Spark 2.0. The public interface does not allow users to unregister the Spark built-in functions. Thus, users will never use Hive's built-in hash function.

The only exception here is TestHiveFunctionRegistry, which allows users to unregister the built-in functions. Thus, we can load Hive's hash function in the test cases. If we disable it, 10+ test cases will fail because the results are different from the Hive golden answer files.

This PR is to remove hash from the list of hiveFunctions in HiveSessionCatalog. It will also remove TestHiveFunctionRegistry. This removal makes us easier to remove TestHiveSessionState in the future.

How was this patch tested?

N/A

[SparkQA]

Test build #63240 has finished for PR 14498 at commit 3b3f3c8.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

retest this please

[SparkQA]

Test build #63247 has finished for PR 14498 at commit 3b3f3c8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

This seems fair. Shouldn't we do something about all the disabled test cases though?

[gatorsmile]

Yeah, we should check all the disabled test cases and see if we can move it. I am waiting for #14472.

: )

[gatorsmile]

#14472 has been merged. Will work on it soon.

[tejasapatil]

Is Spark's hashing function semantically equivalent to Hive's ? AFAIK, its not. I think it would be better to have a mode to be able to use Hive's hash method. eg. case when this would be needed: Users running a query in Hive want to switch to Spark. As this happens, you want to verify if the data produced is same or not. Also, for a brief time the pipeline would run in both the engines. Upstream consumers of the data generated should not see differences due to running in the different engines

[gatorsmile]

@tejasapatil Thanks for the inputs! I can understand your usage cases. It looks very reasonable.

Actually, in Spark 2.0, we already removed it. You are unable to load/invoke the Hive's hash function. This PR is just to clean the usage of hash function in our testing package.

[rxin]

@gatorsmile can we take this to the finish line? This actually does not impact hash at all since it's only here for testing.

[gatorsmile]

@rxin Sure, will do it this weekend. Thanks!

[cloud-fan]

We can still remove this class if we add back the removed spark builtin hash function manually right?

[gatorsmile]

Yeah, I think so

[rxin]

If this is no longer WIP, please update the title. Thanks.

[gatorsmile]

Based on the previous discussions in the other PRs, it sounds like these Hive-specific test cases are not very useful. Do we still need to rewrite them and add them to the other test suites?

    "auto_join19",
    "auto_join22",
    "auto_join25",
    "auto_join26",
    "auto_join27",
    "auto_join28",
    "auto_join30",
    "auto_join31",
    "auto_join_nulls",
    "auto_join_reordering_values",
    "correlationoptimizer1",
    "correlationoptimizer2",
    "correlationoptimizer3",
    "correlationoptimizer4",
    "multiMapJoin1",
    "orc_dictionary_threshold",
    "udf_hash"

BTW, I went over these disabled test cases. I think the values are low.

[rxin]

I'm ok with not having them.

[SparkQA]

Test build #68268 has finished for PR 14498 at commit 05390ad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Merging in master/branch-2.1.