Skip to content

Conversation

@wajda
Copy link
Contributor

@wajda wajda commented May 22, 2018

What changes were proposed in this pull request?

Fixes ClassCastException in the array_position function - SPARK-24350

When calling array_position function with a wrong type of the 1st argument an AnalysisException should be thrown instead of ClassCastException

Example:

select array_position('foo', 'bar')
java.lang.ClassCastException: org.apache.spark.sql.types.StringType$ cannot be cast to org.apache.spark.sql.types.ArrayType
	at org.apache.spark.sql.catalyst.expressions.ArrayPosition.inputTypes(collectionOperations.scala:1398)
	at org.apache.spark.sql.catalyst.expressions.ExpectsInputTypes$class.checkInputDataTypes(ExpectsInputTypes.scala:44)
	at org.apache.spark.sql.catalyst.expressions.ArrayPosition.checkInputDataTypes(collectionOperations.scala:1401)
	at org.apache.spark.sql.catalyst.expressions.Expression.resolved$lzycompute(Expression.scala:168)
	at org.apache.spark.sql.catalyst.expressions.Expression.resolved(Expression.scala:168)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAliases$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveAliases$$assignAliases$1$$anonfun$apply$3.applyOrElse(Analyzer.scala:256)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAliases$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveAliases$$assignAliases$1$$anonfun$apply$3.applyOrElse(Analyzer.scala:252)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)

How was this patch tested?

unit test

@wajda wajda changed the title SPARK-24350 "array_position" error fix [SPARK-24350][SQL] "array_position" error fix May 22, 2018

intercept[AnalysisException] {
Seq(("a string element", "a")).toDF().selectExpr("array_position(_1, _2)")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the error message, too.

@maropu
Copy link
Member

maropu commented May 22, 2018

Please describe more in the description; you'd be better to copy the description there.

@maropu
Copy link
Member

maropu commented May 22, 2018

@ueshin can you trigger tests?

@ueshin
Copy link
Member

ueshin commented May 22, 2018

ok to test.

@SparkQA
Copy link

SparkQA commented May 22, 2018

Test build #90990 has finished for PR 21401 at commit 8e8f623.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wajda
Copy link
Contributor Author

wajda commented May 22, 2018

retest this please

1 similar comment
@maropu
Copy link
Member

maropu commented May 23, 2018

retest this please

@viirya
Copy link
Member

viirya commented May 23, 2018

From PR description, Fixes ClassCastException in the array_position function - this seems like a better PR title than the current one.

@SparkQA
Copy link

SparkQA commented May 23, 2018

Test build #91005 has finished for PR 21401 at commit 8e8f623.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kiszk
Copy link
Member

kiszk commented May 23, 2018

retest this please

@kiszk
Copy link
Member

kiszk commented May 23, 2018

good catch, thanks

@SparkQA
Copy link

SparkQA commented May 23, 2018

Test build #91015 has finished for PR 21401 at commit 8e8f623.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member

ueshin commented May 23, 2018

Jenkins, retest this please.

@ueshin
Copy link
Member

ueshin commented May 23, 2018

LGTM pending Jenkins.

@viirya
Copy link
Member

viirya commented May 23, 2018

LGTM

@SparkQA
Copy link

SparkQA commented May 23, 2018

Test build #91020 has finished for PR 21401 at commit 8e8f623.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wajda wajda changed the title [SPARK-24350][SQL] "array_position" error fix [SPARK-24350][SQL] Fixes ClassCastException in the "array_position" function May 23, 2018
@gatorsmile
Copy link
Member

retest this please

@SparkQA
Copy link

SparkQA commented May 23, 2018

Test build #91045 has finished for PR 21401 at commit 8e8f623.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kiszk
Copy link
Member

kiszk commented May 23, 2018

retest this please

@kiszk
Copy link
Member

kiszk commented May 23, 2018

LGTM

@SparkQA
Copy link

SparkQA commented May 23, 2018

Test build #91058 has finished for PR 21401 at commit 8e8f623.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks! Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants