Skip to content

Conversation

@JeroenWarmerdam
Copy link

The following use case causes an overflow when creating the partitions inside JdbcRDD:

val jdbcRDD = new JdbcRDD(sc, () =>
      DriverManager.getConnection(url, username, password),
      "SELECT id FROM documents WHERE ? <= id AND id <= ?",
      lowerBound = 1131544775L,
      upperBound = 567279358897692673L,
      numPartitions = 20,
      mapRow = r => (r.getLong("id"))
)

This is fixed by swapping division and multiplication in the creation of partitions.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@srowen
Copy link
Member

srowen commented Feb 17, 2015

No, this changes the result! For example, (3 * 4) / 5 == 2 but 3 * (4 / 5) == 0. It's a good point though, but why not just cast i to long? (You will need to open a JIRA for a minor bug.)

@andrewor14
Copy link
Contributor

Hi @JeroenWarmerdam could you open an associated JIRA and add it to the title? See how other PRs are opened.

@srowen
Copy link
Member

srowen commented Feb 20, 2015

This appears to be superseded by #4701 Do you mind closing this PR?

@asfgit asfgit closed this in 46462ff Feb 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants