[MINOR][MLlib][SQL] Clean up unused variables and unused import #14836
Conversation
|
|
||
| /** | ||
| * Create an [[org.apache.spark.Accumulator]] variable of a given type, which tasks can "add" | ||
| * Create an [[org.apache.spark.Accumulator]] variable of a given type, to which tasks can "add" |
There was a problem hiding this comment.
oh right...reverting it...
|
|
||
| val d = sc.parallelize(1 to 20) | ||
| an [Exception] should be thrownBy {d.foreach{x => acc.value = x}} | ||
| an [SparkException] should be thrownBy {d.foreach{x => acc.value = x}} |
There was a problem hiding this comment.
can you change this to
intercept[SparkException] {
d.foreach(x => acc.value = x)
}
There was a problem hiding this comment.
sure, doing it now
|
Test build #64501 has finished for PR 14836 at commit
|
|
I just noticed that there are many other places where 'PropSpec' style is used |
|
Test build #64506 has finished for PR 14836 at commit
|
|
Test build #64504 has finished for PR 14836 at commit
|
|
Jenkins retest this please |
|
Test build #64513 has finished for PR 14836 at commit
|
|
Test build #64522 has finished for PR 14836 at commit
|
| withSQLConf("spark.sql.autoBroadcastJoinThreshold" -> "2") { | ||
| val df = spark.range(100).toDF() | ||
| val join = df.join(df, "id") | ||
| val plan = join.queryExecution.executedPlan |
There was a problem hiding this comment.
I'm not sure this is without side effects, even though clearly the variable isn't used. I don't think you can remove the line, necessarily.
There was a problem hiding this comment.
yes, I look carefully about it and you are right. I'll undo it.
|
I think the other changes are trivial but not wrong. I'd generally not bother with these bitty changes. It's not that they're wrong but that it takes me some time to go think through whether they're valid, and it's probably not worth our time collectively. |
|
Thanks Sean, I got what you mean. I'll avoid such PRs in the future, thanks a lot for the tips. |
|
Test build #64584 has finished for PR 14836 at commit
|
|
Merged to master |
4 participants
What changes were proposed in this pull request?
Clean up unused variables and unused import statements, unnecessary
returnandtoArray, and some more style improvement, when I walk through the code examples.How was this patch tested?
Testet manually on local laptop.