Skip to content

Conversation

@shenh062326
Copy link
Contributor

SortMergeJoin use a ArrayBuffer[InternalRow] to store bufferedMatches, if the join have a lot of rows with the same key, it will throw OutOfMemoryError.
Add a ExternalAppendOnlyArrayBuffer to store bufferedMatches instand of ArrayBuffer.

@SparkQA
Copy link

SparkQA commented Feb 26, 2016

Test build #52032 has finished for PR 11386 at commit b0c00f4.

  • This patch fails RAT tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class ExternalAppendOnlyArrayBuffer[T: ClassTag](

@rxin
Copy link
Contributor

rxin commented Jun 15, 2016

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.

@asfgit asfgit closed this in 1a33f2e Jun 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants