Skip to content

Allow shrinking of indices from a previous major #28076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 4, 2018
Merged

Allow shrinking of indices from a previous major #28076

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion core/src/main/java/org/elasticsearch/index/shard/StoreRecovery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,14 +145,22 @@ boolean recoverFromLocalShards(BiConsumer<String, MappingMetaData> mappingUpdate
void addIndices(final RecoveryState.Index indexRecoveryStats, final Directory target, final Sort indexSort, final Directory[] sources,
final long maxSeqNo, final long maxUnsafeAutoIdTimestamp, IndexMetaData indexMetaData, int shardId, boolean split,
boolean hasNested) throws IOException {

// clean target directory (if previous recovery attempt failed) and create a fresh segment file with the proper lucene version
Lucene.cleanLuceneIndex(target);
assert sources.length > 0;
final int luceneIndexCreatedVersionMajor = Lucene.readSegmentInfos(sources[0]).getIndexCreatedVersionMajor();
new SegmentInfos(luceneIndexCreatedVersionMajor).commit(target);
Copy link
Contributor

@s1monw s1monw Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we copy the first index into the target directory instead of writing a dummy commit point? something like this:

 Directory dir = sources[0];
 for (String file : dir.listAll()) {
   target.copyFrom(dir, file, file, IOContext.DEFAULT);
  }

This way we also apply the same optimizations for hardlinking etc and we are totally safe?

Copy link
Contributor

@jpountz jpountz Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yannick and I discussed this option first, but this needs extra care, for instance to not copy the write lock. It's also a bit more involved if we want to track statistics as well for the first source. In the end it's not clear to me which option is better.

Copy link
Contributor Author

@ywelsch ywelsch Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@s1monw I initially started with the approach that you've outlined, but found it to be more complex for the reasons that @jpountz stated. In the future, we can hopefully create an IndexWriter for an older Lucene version (@jpountz will raise this on the Lucene project).

Copy link
Contributor

@jpountz jpountz Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just opening the issue but I'll wait to see the conclusion here first in case we decide copying the first directory manually is still a better trade-off.

Copy link
Contributor

@s1monw s1monw Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good to me!


final Directory hardLinkOrCopyTarget = new org.apache.lucene.store.HardlinkCopyDirectoryWrapper(target);

IndexWriterConfig iwc = new IndexWriterConfig(null)
.setCommitOnClose(false)
// we don't want merges to happen here - we call maybe merge on the engine
// later once we stared it up otherwise we would need to wait for it here
// we also don't specify a codec here and merges should use the engines for this index
.setMergePolicy(NoMergePolicy.INSTANCE)
.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
.setOpenMode(IndexWriterConfig.OpenMode.APPEND);
if (indexSort != null) {
iwc.setIndexSort(indexSort);
}
Expand Down
67 changes: 67 additions & 0 deletions qa/full-cluster-restart/src/test/java/org/elasticsearch/upgrades/FullClusterRestartIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,73 @@ public void testShrink() throws IOException {
assertEquals(numDocs, totalHits);
}

public void testShrinkAfterUpgrade() throws IOException {
String shrunkenIndex = index + "_shrunk";
int numDocs;
if (runningAgainstOldCluster) {
XContentBuilder mappingsAndSettings = jsonBuilder();
mappingsAndSettings.startObject();
{
mappingsAndSettings.startObject("mappings");
mappingsAndSettings.startObject("doc");
mappingsAndSettings.startObject("properties");
{
mappingsAndSettings.startObject("field");
mappingsAndSettings.field("type", "text");
mappingsAndSettings.endObject();
}
mappingsAndSettings.endObject();
mappingsAndSettings.endObject();
mappingsAndSettings.endObject();
}
mappingsAndSettings.endObject();
client().performRequest("PUT", "/" + index, Collections.emptyMap(),
new StringEntity(mappingsAndSettings.string(), ContentType.APPLICATION_JSON));

numDocs = randomIntBetween(512, 1024);
indexRandomDocuments(numDocs, true, true, i -> {
return JsonXContent.contentBuilder().startObject()
.field("field", "value")
.endObject();
});
} else {
String updateSettingsRequestBody = "{\"settings\": {\"index.blocks.write\": true}}";
Response rsp = client().performRequest("PUT", "/" + index + "/_settings", Collections.emptyMap(),
new StringEntity(updateSettingsRequestBody, ContentType.APPLICATION_JSON));
assertEquals(200, rsp.getStatusLine().getStatusCode());

String shrinkIndexRequestBody = "{\"settings\": {\"index.number_of_shards\": 1}}";
rsp = client().performRequest("PUT", "/" + index + "/_shrink/" + shrunkenIndex, Collections.emptyMap(),
new StringEntity(shrinkIndexRequestBody, ContentType.APPLICATION_JSON));
assertEquals(200, rsp.getStatusLine().getStatusCode());

numDocs = countOfIndexedRandomDocuments();
}

Response rsp = client().performRequest("POST", "/_refresh");
assertEquals(200, rsp.getStatusLine().getStatusCode());

Map<?, ?> response = toMap(client().performRequest("GET", "/" + index + "/_search"));
assertNoFailures(response);
int totalShards = (int) XContentMapValues.extractValue("_shards.total", response);
assertThat(totalShards, greaterThan(1));
int successfulShards = (int) XContentMapValues.extractValue("_shards.successful", response);
assertEquals(totalShards, successfulShards);
int totalHits = (int) XContentMapValues.extractValue("hits.total", response);
assertEquals(numDocs, totalHits);

if (runningAgainstOldCluster == false) {
response = toMap(client().performRequest("GET", "/" + shrunkenIndex + "/_search"));
assertNoFailures(response);
totalShards = (int) XContentMapValues.extractValue("_shards.total", response);
assertEquals(1, totalShards);
successfulShards = (int) XContentMapValues.extractValue("_shards.successful", response);
assertEquals(1, successfulShards);
totalHits = (int) XContentMapValues.extractValue("hits.total", response);
assertEquals(numDocs, totalHits);
}
}

void assertBasicSearchWorks(int count) throws IOException {
logger.info("--> testing basic search");
Map<String, Object> response = toMap(client().performRequest("GET", "/" + index + "/_search"));
Expand Down