[SPARK-17915][SQL] Prepare a new ColumnVector implementation for UnsafeData

[kiszk]

What changes were proposed in this pull request?

This PR prepares a new implementation OnHeapUnsafeColumnarVector that is optimized for reading data from a Unsafe related data structure (e.g. UnsafeArrayData or UnsafeMapData).

Current implementations of ColumnarVector are OnHeapColumnarVector and OffHeapColumnarVector, which are optimized for reading data from Parquet. When they get an array, an map, or a struct stored in a Unsafe related data structure, operations in OnHeapColumnarVector and OffHeapColumnarVector lead to additional copy operations or data conversion. The OnHeapUnsafeColumnarVector only requires a simple memory copy and keep data in an byte array.

OnHeapUnsafeColumnarVector can compress/decompress stored data by using CompressionCodec with spark.sql.inMemoryColumnarStorage.compression.codec property (default is lz4).

This PR is a part of #15219. This is an component independent of others. For ease of review, this PR only introduces OnHeapUnsafeColumnarVector

How was this patch tested?

Perform existing tests for OnHeapUnsafeColumnVector

[SparkQA]

Test build #66901 has finished for PR 15468 at commit 8b38531.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66902 has finished for PR 15468 at commit 9a57147.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66904 has finished for PR 15468 at commit 870c644.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66920 has finished for PR 15468 at commit f6dc8ac.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public final class OnHeapUnsafeColumnVector extends ColumnVector implements Serializable

[kiszk]

@davies, could you please review this at first if #15219 is too big?
cc @tedyu, @sameeragarwal

[kiszk]

@andrewor14 would it be possible to review this?

[kiszk]

@sameeragarwal, would it be possible to review this, too?

[kiszk]

ping @sameeragarwal

[kiszk]

@sameeragarwal, could you please take a look at this?

[SparkQA]

Test build #72864 has finished for PR 15468 at commit de17650.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public final class OnHeapUnsafeColumnVector extends ColumnVector implements Serializable

[SparkQA]

Test build #72886 has finished for PR 15468 at commit 80a5022.

• This patch fails from timeout after a configured wait of `250m`.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

Jenkins, retest this please

[SparkQA]

Test build #72939 has finished for PR 15468 at commit 80a5022.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

@sameeragarwal would it be possible to review this since I resolved a conflict?

[kiszk]

ping @sameeragarwal

[kiszk]

I close this since #18014 and #18033 enable the same feature.