Allow upgrade API to only upgrade too-old segments

[clintongormley]

The upgrade API today will upgrade any segments which are not the latest Lucene version. When upgrading to 2.0, we won't be able to read 3.x segments, but we can read 4.x segments, so it would be faster for the upgrade API to upgrade JUST the 3.x segments, and leave the 4.x segments to the merge process.

[rmuir]

Upgrading only 3.x is short-sighted and will leave the same trap for userse for 2.0->3.0.

I do think only segments that are "out of date" should be upgraded. I do think its also nice to have an option to upgrade only the SUPER ANCIENT ones and maybe some tools and stuff can use that. But it should not be the default behavior.

[rmuir]

I would go further too, if we add an 'ancient' only option (segments from previous lucene major version), and make sure logging and various stats apis etc scream bloody murder when users have ancient segments. Really make the user feel shame and humiliation for still having them around. Otherwise nothing will get done.

[clintongormley]

I do think only segments that are "out of date" should be upgraded. I do think its also nice to have an option to upgrade only the SUPER ANCIENT ones and maybe some tools and stuff can use that. But it should not be the default behavior.

Agreed - it's just about adding the option to do the least work possible to get your cluster moving again. This would be particularly useful when combined with the upgrade-old-segments-as-part-of-background-merge-process functionality that we've discussed.