[SQL][CORE] SPARK-2102

[ianoc]

Adds in an option into the kryo serializer to enable requiring registration

Adds a resource pool in the SparkSqlSerializer code paths to speed up serialization of generic types.

[AmplabJenkins]

Can one of the admins verify this patch?

[marmbrus]

test this please

[SparkQA]

QA tests have started for PR 1377. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16610/consoleFull

[SparkQA]

QA results for PR 1377:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16610/consoleFull

[marmbrus]

@pwendell, any thoughts on the additional option for kryo?

[pwendell]

Hey @ianoc - I found the wording here a bit confusing when reading it over. It would be good to say the exact behavior of Kryo if this is set to true (i.e. will it throw an exception if an unregistered class is found?). Also, the this in the third sentence, it's a bit ambigugous whether that refers to the registration itself or the default setting of this value. A slight re-word would be :

Whether to require registration with Kryo. If set to 'true', Kryo will throw an exception
if an unregistered class is serialized. If set to false (the default), Kryo will write
unregistered class names along with each object. Writing class names can cause 
significant performance overhead, so enabling this option can enforce strictly that a
user has not omitted classes from registration.

[ianoc]

@pwendell Sounds good to me, updated as per your suggestion.

[marmbrus]

test this please

[SparkQA]

QA tests have started for PR 1377. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17062/consoleFull

[SparkQA]

QA results for PR 1377:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
if an unregistered class is serialized. If set to false (the default), Kryo will write

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17062/consoleFull

[marmbrus]

Thanks! I've merged this into master.

[ianoc]

Excellent thanks