[SPARK-5269] [CORE] Use a SerializerInstance pool in BlockManager.

[maciej-kisiel]

Instead of creating a serializer instance each time the BlockManager serialization methods are used, keep a pool of available serializers. It should improve performance.

[mccheah]

Jenkins, test this please

[andrewor14]

ok to test

[andrewor14]

@maciej-kisiel have you done any benchmarking on this?

[SparkQA]

Test build #41892 has finished for PR 8292 at commit a3bddec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maciej-kisiel]

@andrewor14 I'm working on it

[SparkQA]

Test build #42138 has finished for PR 8292 at commit 492c06a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maciej-kisiel]

Weird, I just rebased it..

[andrewor14]

Yeah master tests are flaky. retest this please

[SparkQA]

Test build #42276 has finished for PR 8292 at commit 492c06a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

@maciej-kisiel could you rebase this one?

[andrewor14]

@maciej-kisiel any updates on this one? Do you still plan to work on this?

[maciej-kisiel]

It will be difficult for me to benchmark this, because in the meantime I lost access to a big data environment.

[andrewor14]

I can see how this patch can improve performance; currently we may end up creating a SerializerInstance per partition, which could be expensive if there are thousands of partitions. However, I don't know how much performance this could save in practice. If it doesn't end up buying us much then the extra complexity here is not worth it.

[srowen]

Agree, do you mind closing this PR?