[SPARK-8264][SQL]add substring_index function

[zhichao-li]

No description provided.

[SparkQA]

Test build #37824 has finished for PR 7533 at commit e52b379.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[SparkQA]

Test build #37941 has finished for PR 7533 at commit 4fe5df5.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[SparkQA]

Test build #38006 has finished for PR 7533 at commit 12e108f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[zhichao-li]

cc @chenghao-intel @rxin

[chenghao-intel]

Need to override the nullable method, as the default value is false.

[zhichao-li]

I guess it's abstract in Expression which Substring_index inherited from?

[chenghao-intel]

Sorry, my bad, actually I mean the function foldable, not the nullable.

[zhichao-li]

@chenghao-intel just refactor the code to remedy the problem of invoking numChars to much, but I still think we should add numChars as a field to UTF8String since it's quite an useful function.

[zhichao-li]

@rxin @chenghao-intel should I turn these functions(ordinalIndexOf, lastOrdinalIndexOf and subStringIndex) to be public in UTF8String? I guess they would be useful but on one use them except for substring_index UDF.

[SparkQA]

Test build #38058 has finished for PR 7533 at commit ac863e9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[zhichao-li]

retest this please.

[SparkQA]

Test build #55 has finished for PR 7533 at commit ac863e9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[SparkQA]

Test build #38063 has finished for PR 7533 at commit ac863e9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[chenghao-intel]

Let's remove this version of API. @rxin actually made some clean up, and removed the string (the column name) version API.

[SparkQA]

Test build #38186 has finished for PR 7533 at commit b19b013.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[chenghao-intel]

We'd better not make it the static function, like the indexOf, it mean we are locating the substring from CURRENT string.

[chenghao-intel]

Give more verbose info, like this.toString()?

[SparkQA]

Test build #38294 has finished for PR 7533 at commit 67c253a.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[SparkQA]

Test build #38298 has finished for PR 7533 at commit 9546991.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[chenghao-intel]

We dont' need to check the null value here, as it's done in the expression side.

[SparkQA]

Test build #38317 has finished for PR 7533 at commit 515519b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Substring_index(strExpr: Expression, delimExpr: Expression, countExpr: Expression)

[zhichao-li]

@rxin it looks a few better now. could you take a look at this?

[chenghao-intel]

Nit: throws more concrete exception like IllegalArgumentException or IllegalCharacterException etc.

[chenghao-intel]

LGTM in general, @davies @rxin

[davies]

Could you use TernaryExpression ?

[davies]

@zhichao-li Do you mind me to take over this one?