[SPARK-6775] [SPARK-6776] [SQL] [WIP] Refactors converters for converting Parquet records to row objects

[liancheng]

This PR simplifies the CatalystConverter class hierarchy, and implements backwards-compatibility rules for them. The new class hierarchy basically resembles AvroIndexedRecordConverter in parquet-avro.

This is still in WIP status. I'm opening this PR to check whether it passes Jenkins.

TODO:

• Pass all tests.
• Tests for legacy Parquet data format.

Major changes:

• Replaces CatalystConverter trait with a much simpler ParentContainerUpdater.

  Now instead of extending the original CatalystConverter trait, every converter class accepts an updater whose's responsible to set the converted value to some parent container. For example, appending array elements to a parent array buffer, appending a key-value pairs to a parent mutable map, or setting a converted value to some specific field of a parent row.

  This simplifies the design since converters don't need to care about details of their parent converters anymore.

• Replaces CatalystRootConverter, CatalystGroupConverter and CatalystPrimitiveRowConverter with CatalystRowConverter

  Specifically, now all row objects are represented with SpecificMutableRow during conversion.

• Removes CatalystArrayContainsNullConverter and CatalystNativeArrayConverter, revamped CatalystArrayConverter

  CatalystNativeArrayConverter should be designed with the intention of avoiding boxing costs. However, the way it uses Scala generics actually doesn't achieve this goal.

  The new CatalystArrayConverter handles both nullable and non-nullable array elements.

• Implemented backwards-compatibility rules in CatalystArrayConverter

  When Parquet records are being converted, schema of Parquet files should have already been verified. So we only need to care about the structure rather than field names in the Parquet schema. Since all map objects represented in legacy systems have the same structure as the standard one (see backwards-compatibility rules for MAP), we only need to deal with LIST (namely array) here.

[liancheng]

Dates are represented as integers inside Catalyst. Since SpecificMutableRow is only used inside Catalyst, it should be safe to specialize DateType here.

[SparkQA]

Test build #29858 has finished for PR 5422 at commit 0de2e8b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #29859 has finished for PR 5422 at commit 9522949.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #29868 has finished for PR 5422 at commit 02d78c4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #29869 has finished for PR 5422 at commit 2529b76.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #31021 has finished for PR 5422 at commit 2529b76.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • RoaringBitmap-0.4.5.jar
  • activation-1.1.jar
  • akka-actor_2.10-2.3.4-spark.jar
  • akka-remote_2.10-2.3.4-spark.jar
  • akka-slf4j_2.10-2.3.4-spark.jar
  • aopalliance-1.0.jar
  • arpack_combined_all-0.1.jar
  • avro-1.7.7.jar
  • breeze-macros_2.10-0.11.2.jar
  • breeze_2.10-0.11.2.jar
  • chill-java-0.5.0.jar
  • chill_2.10-0.5.0.jar
  • commons-beanutils-1.7.0.jar
  • commons-beanutils-core-1.8.0.jar
  • commons-cli-1.2.jar
  • commons-codec-1.10.jar
  • commons-collections-3.2.1.jar
  • commons-compress-1.4.1.jar
  • commons-configuration-1.6.jar
  • commons-digester-1.8.jar
  • commons-httpclient-3.1.jar
  • commons-io-2.1.jar
  • commons-lang-2.5.jar
  • commons-lang3-3.3.2.jar
  • commons-math-2.1.jar
  • commons-math3-3.1.1.jar
  • commons-net-2.2.jar
  • compress-lzf-1.0.0.jar
  • config-1.2.1.jar
  • core-1.1.2.jar
  • curator-client-2.4.0.jar
  • curator-framework-2.4.0.jar
  • curator-recipes-2.4.0.jar
  • gmbal-api-only-3.0.0-b023.jar
  • grizzly-framework-2.1.2.jar
  • grizzly-http-2.1.2.jar
  • grizzly-http-server-2.1.2.jar
  • grizzly-http-servlet-2.1.2.jar
  • grizzly-rcm-2.1.2.jar
  • groovy-all-2.3.7.jar
  • guava-14.0.1.jar
  • guice-3.0.jar
  • hadoop-annotations-2.2.0.jar
  • hadoop-auth-2.2.0.jar
  • hadoop-client-2.2.0.jar
  • hadoop-common-2.2.0.jar
  • hadoop-hdfs-2.2.0.jar
  • hadoop-mapreduce-client-app-2.2.0.jar
  • hadoop-mapreduce-client-common-2.2.0.jar
  • hadoop-mapreduce-client-core-2.2.0.jar
  • hadoop-mapreduce-client-jobclient-2.2.0.jar
  • hadoop-mapreduce-client-shuffle-2.2.0.jar
  • hadoop-yarn-api-2.2.0.jar
  • hadoop-yarn-client-2.2.0.jar
  • hadoop-yarn-common-2.2.0.jar
  • hadoop-yarn-server-common-2.2.0.jar
  • ivy-2.4.0.jar
  • jackson-annotations-2.4.0.jar
  • jackson-core-2.4.4.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-databind-2.4.4.jar
  • jackson-jaxrs-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • jackson-module-scala_2.10-2.4.4.jar
  • jackson-xc-1.8.8.jar
  • jansi-1.4.jar
  • javax.inject-1.jar
  • javax.servlet-3.0.0.v201112011016.jar
  • javax.servlet-3.1.jar
  • javax.servlet-api-3.0.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar
  • jcl-over-slf4j-1.7.10.jar
  • jersey-client-1.9.jar
  • jersey-core-1.9.jar
  • jersey-grizzly2-1.9.jar
  • jersey-guice-1.9.jar
  • jersey-json-1.9.jar
  • jersey-server-1.9.jar
  • jersey-test-framework-core-1.9.jar
  • jersey-test-framework-grizzly2-1.9.jar
  • jets3t-0.7.1.jar
  • jettison-1.1.jar
  • jetty-util-6.1.26.jar
  • jline-0.9.94.jar
  • jline-2.10.4.jar
  • jodd-core-3.6.3.jar
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar
  • jsr305-1.3.9.jar
  • jtransforms-2.4.0.jar
  • jul-to-slf4j-1.7.10.jar
  • kryo-2.21.jar
  • log4j-1.2.17.jar
  • lz4-1.2.0.jar
  • management-api-3.0.0-b012.jar
  • mesos-0.21.0-shaded-protobuf.jar
  • metrics-core-3.1.0.jar
  • metrics-graphite-3.1.0.jar
  • metrics-json-3.1.0.jar
  • metrics-jvm-3.1.0.jar
  • minlog-1.2.jar
  • netty-3.8.0.Final.jar
  • netty-all-4.0.23.Final.jar
  • objenesis-1.2.jar
  • opencsv-2.3.jar
  • oro-2.0.8.jar
  • paranamer-2.6.jar
  • parquet-column-1.6.0rc3.jar
  • parquet-common-1.6.0rc3.jar
  • parquet-encoding-1.6.0rc3.jar
  • parquet-format-2.2.0-rc1.jar
  • parquet-generator-1.6.0rc3.jar
  • parquet-hadoop-1.6.0rc3.jar
  • parquet-jackson-1.6.0rc3.jar
  • protobuf-java-2.4.1.jar
  • protobuf-java-2.5.0-spark.jar
  • py4j-0.8.2.1.jar
  • pyrolite-2.0.1.jar
  • quasiquotes_2.10-2.0.1.jar
  • reflectasm-1.07-shaded.jar
  • scala-compiler-2.10.4.jar
  • scala-library-2.10.4.jar
  • scala-reflect-2.10.4.jar
  • scalap-2.10.4.jar
  • scalatest_2.10-2.2.1.jar
  • slf4j-api-1.7.10.jar
  • slf4j-log4j12-1.7.10.jar
  • snappy-java-1.1.1.6.jar
  • spark-bagel_2.10-1.4.0-SNAPSHOT.jar
  • spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
  • spark-core_2.10-1.4.0-SNAPSHOT.jar
  • spark-graphx_2.10-1.4.0-SNAPSHOT.jar
  • spark-launcher_2.10-1.4.0-SNAPSHOT.jar
  • spark-mllib_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-common_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
  • spark-repl_2.10-1.4.0-SNAPSHOT.jar
  • spark-sql_2.10-1.4.0-SNAPSHOT.jar
  • spark-streaming_2.10-1.4.0-SNAPSHOT.jar
  • spire-macros_2.10-0.7.4.jar
  • spire_2.10-0.7.4.jar
  • stax-api-1.0.1.jar
  • stream-2.7.0.jar
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar
  • uncommons-maths-1.2.2a.jar
  • unused-1.0.0.jar
  • xmlenc-0.52.jar
  • xz-1.0.jar
  • zookeeper-3.4.5.jar

[liancheng]

Closing this as it's superceded by #6617 and upcoming follow-up PRs.