Automatically slice reindex and friends

[nik9000]

Background:

Right now reindex and friends have a slices parameter that you can set to automatically parallelize the process. Setting the right value for the slices parameter is a little involved but we have documented guidelines.

Question:

Is it worth implementing something like ?slices=auto? We can do a fairly good job picking the number of slices with something simple like:

if (sourceIndex.numberOfShards > 50) {
  slices = 1; // too big, user should optimize it by hand.
} else {
  slices = sourceIndex.numberOfShards;
}

The point here is that ?slices=auto can be a sane default. Maybe, even, it could be the default in 6.0.

[jpountz]

+1 to a sane default value

Maybe we could default to something like min(numShards, 10), I don't like suddenly deoptimizing when crossing a boundary?

Also I don't think we need a parameter value for it, it could just be the default and users would override it with an explicit value if it does not satisfy their needs?

[nik9000]

Also I don't think we need a parameter value for it, it could just be the default and users would override it with an explicit value if it does not satisfy their needs?

That is fine with me.

Maybe we could default to something like min(numShards, 10), I don't like suddenly deoptimizing when crossing a boundary?

Me neither. The trouble is that it is a bit complex to come up with a good choice if the number of shards is large. I can certainly run some benchmarks to do that though. That might be a good exercise in general though.

[jimczi]

Me neither. The trouble is that it is a bit complex to come up with a good choice if the number of shards is large. I can certainly run some benchmarks to do that though. That might be a good exercise in general though.

The reasoning behind min(numShards, N) is to always pick a number that is less or equals to the number of shards. This way the slicing does not add any overhead since each shard is assigned to a single slice.

[nik9000]

This way the slicing does not add any overhead since each shard is assigned to a single slice.

This is something I hadn't realized. I went and reread that code. I'd thought that had special cases for when shards == slices and shards == <some integer> * slices. But it looks like the special case is shards <= slices and the shards == <some integer> * slices is handled by the general case.

Basically I thought we made more of an effort to keep the slices sized consistently at the cost of efficiency. Instead we go with the most efficient query at the cost of potentially vastly different slice sizes. I'm sure @jpountz and @jimczi know all this but I figured it might be nice to call it out for anyone who ends up on this issue.

In that case min(numShards, N) is a very safe bet.

[andyb-elastic]

@nik9000 I'd like to give this one a shot if it's not too far over my head. I think I follow the discussion here and the docs on slices. Few questions:

1. I'm reading "and friends" here as "APIs that support slices", is that right?
2. Is the benchmarks project the right place to get started with benchmarking/profiling how well the strategy we choose works
3. If (and I may be misunderstanding here) it's preferable to have #slices be a whole factor of #shards, would it make sense to do something like:

choose our constant ceiling, say C = 10
if shards < C
  slices = shards
else
  slices = the largest factor of shards that's under C
endif

[nik9000]

I'm reading "and friends" here as "APIs that support slices", is that right?

update-by-query and delete-by-query. Basically the APIs that are implemented in the reindex module. They all share a ton of code.

Is the benchmarks project the right place to get started with benchmarking/profiling how well the strategy we choose works

I think it is worth talking to @danielmitterdorfer about using https://github.com/elastic/rally for this. It is designed for generating benchmarks. It makes http://benchmarks.elastic.co. Having a reindex benchmark might be a thing. If we had one we could point to this as a big win on the graph. Hopefully.

If (and I may be misunderstanding here) it's preferable to have #slices be a whole factor of #shards, would it make sense to do something like:

That is what I thought at first but isn't true. Sliced scroll runs in two ways:

1. If the number of slices <= shards then assign each shard to a slice and query those shards.
2. Otherwise, assign each shard to a slice and add a filter.

So we want to shoot for slices == shards exactly. Going above makes the query less efficient and isn't obviously better. Maybe in some cases it is but I don't know that it is worth doing by default. But slices <= shards has very low cost which is why we were ok with doing it by default.

That ceiling constant is more about safety than performance. If you set the number of slices too high then reindex can take a ton of memory.

[danielmitterdorfer]

Is the benchmarks project the right place to get started with benchmarking/profiling how well the strategy we choose works

The benchmarks project is about microbenchmarks (benchmarking components of Elasticsearch in isolation), Rally is about macrobenchmarks (end-to-end benchmark of the whole system)

I think it is worth talking to @danielmitterdorfer about using https://github.com/elastic/rally for this.

The Reindex API is not supported out of the box by Rally but it is very easy to add this capability (see the "runner" concept in the Rally docs) and I can definitely help out with that.

Having a reindex benchmark might be a thing. If we had one we could point to this as a big win on the graph. Hopefully.

As a first step, we could definitely add reindex to one of the standard Rally tracks (think: benchmark scenarios). I suggest we add this to the "geonames" track as a starter.

At the moment we're a bit tight on runtime budget in the nightly benchmarks (total duration is currently ~ 15 hours per day) but I am sure we can do something to make it happen midterm.

[nik9000]

At the moment we're a bit tight on runtime budget in the nightly benchmarks (total duration is currently ~ 15 hours per day) but I am sure we can do something to make it happen midterm.

We're getting close to those being "daily" instead of nightly....

[andyb-elastic]

That ceiling constant is more about safety than performance. If you set the number of slices too high then reindex can take a ton of memory

Gotcha, and it seems like a benefit from the user's side is that it gives them a good way to use slices without having to make guesses about how many to configure, and min(numShards, N) won't give any surprises.

As a first step, we could definitely add reindex to one of the standard Rally tracks (think: benchmark scenarios). I suggest we add this to the "geonames" track as a starter.

Sounds good, I'll plan on using geonames. Rally looks like a really cool benchmarking project. I'm sure I'll have more questions for you when I get that far.

[nik9000]

Gotcha, and it seems like a benefit from the user's side is that it gives them a good way to use slices without having to make guesses about how many to configure, and min(numShards, N) won't give any surprises.

Right. I see this as "we are comfortable enough with slices that we may as well always use them when they're obviously faster". The user can get the old behavior back with slices=1.

I think a few PRs worth of changes might come from this issue:

1. Update the guidelines I linked in the description to align with the discussion above.
2. Implement slices=auto.
3. Set it as the default.

I think we're on board with doing all three though I think making auto the default should wait a bit because setting slices to anything but 1 doesn't work with reindex-from-remote. We should probably get it to work with reindex-from-remote. I didn't implement slices against remote nodes because it didn't seem worth it at the time. Now that slice support has been out for longer there will be more opportunities to reindex remotely.