Skip to content

Commit dbcc250

Browse files
committed
metastore: skip checkpoint log when beginning
This log is written on every partition, even when there's no data loss. The position.cmp(&delta_position.from) comparison doesn't account for Position::Beginning being not equal to Position::offset(0u64). However logically speaking moving from Beginning to 0u64 doesn't indicate any data loss - Position::Beginning means that no data was processed, and Position::offset(0) means offset 0 was processed. I have run 'test_process_existing_messages' at quickwit/quickwit-indexing/src/source/queue_sources/coordinator.rs and confirmed that the delta is - SourceCheckpointDelta[Position::offset(0)..Position::eof(file_len)] and not - SourceCheckpointDelta[Position::Beginning..Position::eof(file_len)]
1 parent a58b6de commit dbcc250

File tree

1 file changed

+5
-1
lines changed

1 file changed

+5
-1
lines changed

quickwit/quickwit-metastore/src/checkpoint.rs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,11 @@ impl SourceCheckpoint {
293293
match position.cmp(&delta_position.from) {
294294
Ordering::Equal => {}
295295
Ordering::Less => {
296-
warn!(cur_pos=?position, delta_pos_from=?delta_position.from,partition=?delta_partition, "some positions were skipped");
296+
if !(position == Position::Beginning
297+
&& delta_position.from == Position::offset(0u64))
298+
{
299+
warn!(cur_pos=?position, delta_pos_from=?delta_position.from,partition=?delta_partition, "some positions were skipped");
300+
}
297301
}
298302
Ordering::Greater => {
299303
return Err(IncompatibleCheckpointDelta {

0 commit comments

Comments
 (0)