[SPARK-10035][SQL] Parquet filters does not process EqualNullSafe filter.

[HyukjinKwon]

As I talked with Cheng,

1. I added EquelNullSafe to ParquetFilters
   • It uses the same equality comparison filter with EqualTo since the Parquet filter performs actually null-safe equality comparison.
2. Updated the test code (ParquetFilterSuite)
   • Convert catalyst.Expression to sources.Filter
   • Removed Cast since only Literal is picked up as a proper Filter in DataSourceStrategy
   • Added EquelNullSafe comparison
3. Removed deprecated createFilter for catalyst.Expression

[liancheng]

Please rename maybeAnalyzedPredicate to analyzedPredicates since it's now a Seq[Expression]. Prefix maybe is for Option[T] by convention.

[liancheng]

ok to test

[SparkQA]

Test build #41185 has finished for PR 8275 at commit e5f556b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41204 has finished for PR 8275 at commit 208411a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StringIndexerModel (
  • class ElementwiseProduct(JavaTransformer, HasInputCol, HasOutputCol):
  • implicit class StringToColumn(val sc: StringContext)
  • case class FilterNode(condition: Expression, child: LocalNode) extends UnaryLocalNode
  • abstract class LocalNode extends TreeNode[LocalNode]
  • abstract class LeafLocalNode extends LocalNode
  • abstract class UnaryLocalNode extends LocalNode
  • case class ProjectNode(projectList: Seq[NamedExpression], child: LocalNode) extends UnaryLocalNode
  • case class SeqScanNode(output: Seq[Attribute], data: Seq[InternalRow]) extends LeafLocalNode
  • public final class UTF8String implements Comparable<UTF8String>, Externalizable

[HyukjinKwon]

Could you merge this :) ?

[liancheng]

Thanks! Merging to master (not to branch-1.5 at this moment since this is a new feature).

[liancheng]

I backported this PR to branch-1.5 to fix 1.5 Jenkins build failures. Didn't merge this PR to branch-1.5 before because it introduced new feature after branch-1.5 being cut. This PR also refactored ParquetFilterSuite and cleaned up outdated code path used by the old Parquet support which has already been removed. The problem is that, PR #8403 merged later depends on the test suite refactoring to pass the build. That's why #8403 passes for master but breaks branch-1.5.

This backporting should be safe, since the EqualSafeNull support has already been merged into branch-1.5 by #8403 (probably done by Git automatically and unintentionally). So this backporting only introduces test suite changes. See changed files of this backporting commit.

cc @pwendell @rxin @JoshRosen