Skip to content

Add lower bound for translog generation threshold #23779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 28, 2017
Merged

Add lower bound for translog generation threshold #23779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions core/src/main/java/org/elasticsearch/index/IndexSettings.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,15 @@ public final class IndexSettings {
Setting.byteSizeSetting(
"index.translog.generation_threshold_size",
new ByteSizeValue(64, ByteSizeUnit.MB),
/*
* An empty translog occupies 43 bytes on disk. If the generation threshold is
* below this, the flush thread can get stuck in an infinite loop repeatedly
* rolling the generation as every new generation will already exceed the
* generation threshold. However, small thresholds are useful for testing so we
* do not add a large lower bound here.
*/
new ByteSizeValue(64, ByteSizeUnit.BYTES),
Copy link
Contributor

@bleskes bleskes Mar 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should wire this to org.elasticsearch.index.translog.TranslogWriter#getHeaderLength(int) with a static constant somewhere.

Copy link
Member Author

@jasontedor jasontedor Mar 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good thought, and I thought about it after your suggestion. I would prefer to keep this predictable for now rather than having the minimum quietly change if we change the translog. I do not expect the header to be changing frequently, but we can revisit this if needed.

new ByteSizeValue(Long.MAX_VALUE, ByteSizeUnit.BYTES),
new Property[]{Property.Dynamic, Property.IndexScope});

public static final Setting<TimeValue> INDEX_SEQ_NO_CHECKPOINT_SYNC_INTERVAL =
Expand Down
5 changes: 3 additions & 2 deletions core/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ public void testMaybeFlush() throws Exception {
}

public void testMaybeRollTranslogGeneration() throws Exception {
final int generationThreshold = randomIntBetween(1, 512);
final int generationThreshold = randomIntBetween(64, 512);
final Settings settings =
Settings
.builder()
Expand All @@ -380,7 +380,8 @@ public void testMaybeRollTranslogGeneration() throws Exception {
int rolls = 0;
final Translog translog = shard.getEngine().getTranslog();
final long generation = translog.currentFileGeneration();
for (int i = 0; i < randomIntBetween(32, 128); i++) {
final int numberOfDocuments = randomIntBetween(32, 128);
Copy link
Contributor

@bleskes bleskes Mar 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

for (int i = 0; i < numberOfDocuments; i++) {
assertThat(translog.currentFileGeneration(), equalTo(generation + rolls));
final ParsedDocument doc = testParsedDocument(
"1",
Expand Down