[SPARK-14859][PYSPARK] Make Lambda Serializer Configurable

[njwhite]

What changes were proposed in this pull request?

Store the serializer that we should use to serialize RDD transformation
functions on the SparkContext, defaulting to a CloudPickleSerializer if not
given. Allow a user to change this serializer when first constructing the
SparkContext.

How was this patch tested?

Unit tests and manual integration tests.

[AmplabJenkins]

Can one of the admins verify this patch?

[holdenk]

Is this functionality we want to add? cc @davies ?

[holdenk]

If we do end up adding this we would probably want to add a test of using a custom serializer (but maybe don't rush to do this since I think if we want to expose this is maybe not yet clear).

[davies]

@njwhite We still use PickleSerializer to deserialize the functions, so it means the serializer MUST be compatible with Pickle, I'm not sure make it configurable will be really helpful (not a good API interface).

If you really want to hack it in your case, I think you could have many ways to hack it in Python.

[njwhite]

@davies I'm using this to use the "dill" serializer, as it can pickle more things (and allows more fine-grained control) than the cloud-pickle serializer. What about making that the default for functions?

[davies]

Can we support dill directly and have a flag to choose from the two serializer? cloud-pickler could be the default one.

[holdenk]

I don't see much progress around this - would it maybe make sense to close and just focus on improving cloudpickle (or upgrading our cloudpickle)?

[HyukjinKwon]

@njwhite It seems inactive for few months. Would this be better to close this for now if you are currently not able to proceed this further?