[SPARK-10417][SQL] Iterating through Column results in infinite loop

[0x0FFF]

pyspark.sql.column.Column object has __getitem__ method, which makes it iterable for Python. In fact it has __getitem__ to address the case when the column might be a list or dict, for you to be able to access certain element of it in DF API. The ability to iterate over it is just a side effect that might cause confusion for the people getting familiar with Spark DF (as you might iterate this way on Pandas DF for instance)

Issue reproduction:

df = sqlContext.jsonRDD(sc.parallelize(['{"name": "El Magnifico"}']))
for i in df["name"]: print i

[srowen]

Don't know much about Python myself but that sounds convincing. CC @davies

[cloud-fan]

you can use assertRaises to test the exception case.

[SparkQA]

Test build #1712 has finished for PR 8574 at commit ea2e9d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class JavaTrainValidationSplitExample
  • class KMeans @Since("1.5.0") (
  • class DCT(JavaTransformer, HasInputCol, HasOutputCol):
  • class SQLTransformer(JavaTransformer):
  • class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):
  • case class LimitNode(limit: Int, child: LocalNode) extends UnaryLocalNode
  • case class UnionNode(children: Seq[LocalNode]) extends LocalNode

[0x0FFF]

@cloud-fan, I addressed your comments with last commit

[0x0FFF]

Looks like it's not being retested after the last commit as Jenkins failed to update the status and the dashboard shows that it's still running. Am I right?

[0x0FFF]

Jenkins, retest this please

[davies]

LGTM

[SparkQA]

Test build #1714 has finished for PR 8574 at commit f041635.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

Merged into master, thanks!