Is disabling refresh a good idea?

[GlenRSmith]

I've recently encountered a heavy logging use case where a single node among about 10 data nodes had extremely high bulk rejection rates and GC issues.

It turned out that all of the refresh interval settings were completely dependent on the node setting, and the one node experiencing issues was set to "-1".

Adding the intentional refresh interval to the index templates remediated the problem.

Is it still the case that there are scenarios in which disabling refresh is beneficial on the whole? Would it be worth considering removing the ability to do so? If so, then the same thing could be accomplished by setting it to an arbitrarily large time span. Perhaps that should have a max value?

@clintongormley

Elasticsearch version: 2.3.4

[s1monw]

first of all you can't set this anymore on a node level in 5.0 - that's a good thing. That said, I am really surprised that not running refresh prevents rejections? Why did documents get rejected in the first place, why would refresh help here?

[clintongormley]

If I'm not mistake, I've heard recently about disabling refresh completely resulting in heavier GC than just using a refresh interval of (eg) 30s. @mikemccand am I remembering correctly?

[GlenRSmith]

@s1monw I think that what was happening was that on the node in question, fully consuming the indexing buffer before refreshing led to high GCs, and the GC activity led to high bulk rejections.

Just to give you an idea of the magnitude of the impact, the largest daily index in the cluster had something like 32 primary shards (warranted by the size of the index 500 - 800GB). Any shard which had a copy on the affected node was approximately 20% the size of every other shard in the index.

Once that particular configuration was removed, the problem went away.

I thought that it was a fascinating opportunity, an experiment that I probably would never have actually set up myself, to demonstrate the contrast between the behavior of bulk indexing with that single difference in configuration.

[s1monw]

I really wonder if the live-version map is involved in this @mikemccand ?

[s1monw]

I also wonder if 5.0 still has this problem?

[mikemccand]

Disabling refresh should not hurt anything, and I don't think I've seen that in the past. @clintongormley do you remember the context before?

It should rather be the opposite: refresh itself is a costly op, forcing a full write of all buffered docs to files, opening readers, etc.

@s1monw we fixed ES a while back to force a refresh if version map was too large, so I don't think it should be due to version map usage.

If RAM usage tracking is working correctly, when you tell ES how much RAM it can use for the indexing buffer, it will write things to disk once they exceed that much RAM. Is is possible the customer here has too large an indexing buffer set?

[s1monw]

@mikemccand I had a conversation with @GlenRSmith but we didn't really find anything in particular. What wasn't clear to me here was that in that cluster that it happened it was only a single node that behaved odd. That node hat the refresh interval set to -1 in the elasticsearch.yaml all other had it set to 30s or so. So I think it has something todo with this. One idea that came up was that it wasn't the GCs causing the rejections but rather an unlucky situation where we have a reader open to look up versions that has a very very large number of segments and we don't refresh it for a long time. I wonder if it's worth it to enhance IMC to also cause a refresh if there are merged segments ready. I can see that this can cause slow lookups and slowing indexing down dramatically if we have a reader that is so out of date. does this make sense?

[GlenRSmith]

@s1monw @clintongormley @mikemccand from Simon's mention of version maps, I took a look and found that the node in question had a 268 MB version map. Other data-only nodes had version maps ranging from 5.5 - 8.6 MB.