KAFKA-19407 Fix potential IllegalStateException when appending to timeIndex #19972
Conversation
github-actions
bot
added
triage
|
This PR is supposed to introduce no any new lock contention, because (potentially blocking) lazy timeIndex materialization has already exclusive control by lock in LazyIndex#get |
| @@ -192,8 +192,13 @@ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException { | |||
| * the time index). | |||
| */ | |||
| public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException { | |||
| if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN) | |||
| maxTimestampAndOffsetSoFar = timeIndex().lastEntry(); | |||
| if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN) { | |||
There was a problem hiding this comment.
Should this volatile field become an Atomic instead?
There was a problem hiding this comment.
Using Atomic doesn't make the thing simple, because we need double-check locking anyways to ensure:
- initialize the value only once at first time
- always return initialized value
There was a problem hiding this comment.
I thought updateAndGet could work to this effect
maxTimestampAndOffsetSoFar.updateAndGet(t -> if (t == TimestampOffset.UNKNOWN) timeIndex().lastEntry() else t)But I have not fully considered whether it would end up being slower on any microbenchmark. It just looked simpler.
| @@ -192,8 +192,13 @@ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException { | |||
| * the time index). | |||
| */ | |||
| public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException { | |||
| if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN) | |||
There was a problem hiding this comment.
Perhaps we can prevent readMaxTimestampAndOffsetSoFar from changing maxTimestampAndOffsetSoFar by initializing the latter with timeIndex().lastEntry() in the constructor?
There was a problem hiding this comment.
If we do so, memory map for all segments will be created eagerly at startup and make it slow against the motivation of lazy index loading
There was a problem hiding this comment.
memory map for all segments will be created eagerly at startup
You're right.
The lock approach works for me, but maybe there's another approach that doesn't need a lock. We can pre-load the maxTimestampAndOffsetSoFar at startup for the active segment. If I understand this issue correctly, the issue happens only in the active segment. Fortunately, the memory map of the active segment is always pre-loaded. Hence, we can initialize the maxTimestampAndOffsetSoFar at the same time. WDYT?
Summary