Skip to content

[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in ConnectionManager #2742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in ConnectionManager #2742

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
52b6cd2
[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in Connect…
aarondav Oct 10, 2014
a152065
Add other pathway back
aarondav Oct 10, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,27 @@ sealed abstract class ManagedBuffer {
final class FileSegmentManagedBuffer(val file: File, val offset: Long, val length: Long)
extends ManagedBuffer {

/**
* Memory mapping is expensive and can destabilize the JVM (SPARK-1145, SPARK-3889).
* Avoid unless there's a good reason not to.
*/
private val MIN_MEMORY_MAP_BYTES = 2 * 1024 * 1024;

override def size: Long = length

override def nioByteBuffer(): ByteBuffer = {
var channel: FileChannel = null
try {
channel = new RandomAccessFile(file, "r").getChannel
channel.map(MapMode.READ_ONLY, offset, length)
// Just copy the buffer if it's sufficiently small, as memory mapping has a high overhead.
if (length < MIN_MEMORY_MAP_BYTES) {
val buf = ByteBuffer.allocate(length.toInt)
channel.read(buf, offset)
buf.flip()
buf
} else {
channel.map(MapMode.READ_ONLY, offset, length)
}
} catch {
case e: IOException =>
Try(channel.size).toOption match {
Expand Down