Should reindex-from-remote allow indexing lots of indices into local indexes with the same name?

[nik9000]

Right now reindex-from-remote only indexes into a single destination index by default. You can use this technique to make it write into indices with a different name, but you can't get it to write into exactly the same index name. The script doesn't support that. Should we do something to make reindex-from-remote support writing to the same target indexes?

[nik9000]

Personally I wouldn't want to use this feature. If I had many indices that I wanted to reindex-from-remote I'd write a bash script to reindex them one by one. That way I could see progress and parallelize them or whatever. I've been known to write some pretty fancy (still horrible) bash scripts for this kind of thing....

[clintongormley]

I like the ease of use of reindexing all indices to a new cluster, but I think i'm persuaded by your argument:

If I had many indices that I wanted to reindex-from-remote I'd write a bash script to reindex them one by one. That way I could see progress and parallelize them or whatever.

[nik9000]

Talked with a small group - I'll update the docs with the bash script I'd write.

[ruflin]

@nik9000 Did the bash script end up in the docs?

The use case we just had for this is migrating a local Elasticsearch cluster with daily indices to Elastic Cloud. I personally think there would be value in making this use case as simple as possible and not require an external script to work.

What is the reason the script works with a new index but fails with the exact same one?

[nik9000]

@nik9000 Did the bash script end up in the docs?

No. I forgot about this issue.

The nice thing about an external script is that it can be made to be fairly pause-able and resume-able, both of which are harder with reindex.

If you want to dump many external indices into the same target index that'll work. But reindex doesn't have a "same_as_source" option. You can use a script to get it, but I figure if you are loading many indices you are better of doing it with something you have more control over.

[dpavlos]

Hi there,

in case anyone else stumble upon this, here is a very simple python script I've used to migrate our ES to v5. The script pretty much follows @nik9000 idea.

[bleskes]

@nik9000 are you still planning to do this? if not, it's fine. Please close the issue.