[SPARK-12598][Core] bug in setMinPartitions

[datafarmer]

There is a bug in the calculation of maxSplitSize. The totalLen should be divided by minPartitions and not by files.size.

[srowen]

Agree, compare to the impl in WholeTextInputFormat. Really it can be tidier, and fix minPartitions = 0, with:

  def setMinPartitions(context: JobContext, minPartitions: Int) {
    val totalLen = listStatus(context).asScala.filterNot(_.isDir).map(_.getLen).sum
    val maxSplitSize = math.ceil(totalLen / math.max(minPartitions, 1.0)).toLong
    super.setMaxSplitSize(maxSplitSize)
  }

But @datafarmer please see https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark for how we suggest changes first.

@kmader WDYT?

[datafarmer]

@srowen I guess that I should have created a JIRA ticket first. I just created one: SPARK-12598

[srowen]

@datafarmer go ahead and update the title here and consider updating the PR itself per above.

[datafarmer]

@srowen I'll update the PR per your changes. BTW, the FileStatus method isDir is deprecated. Should I change it to isDirectory, or is that something for another PR?

[srowen]

@datafarmer I've just seconds ago merged a change that replaces these deprecated calls, since we can assume Hadoop 2.2+ now. Yes, isDirectory is correct now.

[srowen]

@datafarmer are you able to update this?

[datafarmer]

@srowen It should already be updated per your request. Let me know if there is something else that needs to be done.

[srowen]

@datafarmer this still shows a merge conflict though. That's what needs to be resolved with a rebase.

[datafarmer]

@srowen Should be OK now.

[SparkQA]

Test build #2347 has finished for PR 10546 at commit 73b0e0b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/1.6