[SPARK-26004][SQL] InMemoryTable support StartsWith predicate push down

[wangyum]

What changes were proposed in this pull request?

SPARK-24638 adds support for Parquet file StartsWith predicate push down.
InMemoryTable can also support this feature.

This is an example to explain how it works, Imagine that the id column stored as below:

Partition ID	lowerBound	upperBound
p1	'1'	'9'
p2	'10'	'19'
p3	'20'	'29'
p4	'30'	'39'
p5	'40'	'49'

A filter df.filter($"id".startsWith("2")) or id like '2%'
then we substr lowerBound and upperBound:

Partition ID	lowerBound.substr(0, Length("2"))	upperBound.substr(0, Length("2"))
p1	'1'	'9'
p2	'1'	'1'
p3	'2'	'2'
p4	'3'	'3'
p5	'4'	'4'

We can see that we only need to read p1 and p3.

How was this patch tested?

unit tests and benchmark tests

benchmark test result:

================================================================================================
Pushdown benchmark for StringStartsWith
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '10%'): Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
InMemoryTable Vectorized                    12068 / 14198          1.3         767.3       1.0X
InMemoryTable Vectorized (Pushdown)           5457 / 8662          2.9         347.0       2.2X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '1000%'): Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
InMemoryTable Vectorized                      5246 / 5355          3.0         333.5       1.0X
InMemoryTable Vectorized (Pushdown)           2185 / 2346          7.2         138.9       2.4X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '786432%'): Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
InMemoryTable Vectorized                      5112 / 5312          3.1         325.0       1.0X
InMemoryTable Vectorized (Pushdown)           2292 / 2522          6.9         145.7       2.2X

[SparkQA]

Test build #98689 has finished for PR 23004 at commit 7bbdb07.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

cc @cloud-fan @HyukjinKwon @kiszk

[cloud-fan]

can you add some comment to explain it?

[wangyum]

Added to pr description.

[maropu]

Can you add the comment in the line 240, too?

[wangyum]

@maropu Done

[HyukjinKwon]

BTW, a.startswith(b) and b.startswith(a) are not same but why are they same here?

[cloud-fan]

same question

[wangyum]

Good question, The last one should be removed, DataSourceStrategy has the same logic:

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala

Lines 512 to 513 in 3d6b68b

	case expressions.StartsWith(a: Attribute, Literal(v: UTF8String, StringType)) =>
	Some(sources.StringStartsWith(a.name, v.toString))

[HyukjinKwon]

Looks fine to me

[SparkQA]

Test build #99040 has finished for PR 23004 at commit 0748deb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

Any update?

[SparkQA]

Test build #103146 has finished for PR 23004 at commit 15b43f9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

cc: @cloud-fan @HyukjinKwon

[cloud-fan]

thanks, merging to master!