VersionUtilsTest#testGradleVersionsMatchVersionUtils fails

[danielmitterdorfer]

With #27778 we changed the logic what versions are classified as snapshot versions (by considering the value of build.snapshot). As a consequence, VersionUtilsTest#testGradleVersionsMatchVersionUtils is now failing if a build runs with -Dbuild.snapshot=false.

Reproduction line: gradle :test:framework:test -Dtests.class=org.elasticsearch.test.VersionUtilsTests -Dtests.method="testGradleVersionsMatchVersionUtils" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false"

Fails with:

08:19:52 FAILURE 0.02s J0 | VersionUtilsTests.testGradleVersionsMatchVersionUtils <<< FAILURES!
08:19:52    > Throwable #1: java.lang.AssertionError: expected:<[6.0.0, 6.0.1]> but was:<[6.0.0, 6.0.1, 6.0.2, 6.1.0, 6.2.0]>
08:19:52    > 	at __randomizedtesting.SeedInfo.seed([D4465A3589E8FF0E:185403044F8B2C0]:0)
08:19:52    > 	at org.elasticsearch.test.VersionUtilsTests.testGradleVersionsMatchVersionUtils(VersionUtilsTests.java:278)
08:19:52    > 	at java.lang.Thread.run(Thread.java:748)

(taken from elastic+elasticsearch+master+release-tests build #289).

So the problem is that 6.0.2, 6.1.0, 6.2.0 were considered unreleased at that point in time (the version 6.1.0 is already released but by the time the build ran, the Versions.java has not yet been adjusted) but still appeared as released.

[danielmitterdorfer]

I've muted the test now in case -Dbuild.snapshot is false on 6.1, 6.x and master. See above for the respective commits.

[danielmitterdorfer]

I've now also muted the test on 6.0 (missed it before).

[jasontedor]

For future reference, a better way to mute this is to use assumeTrue; then the test shows up as silenced:

diff --git a/test/framework/src/test/java/org/elasticsearch/test/VersionUtilsTests.java b/test/framework/src/test/java/org/elasticsearch/test/VersionUtilsTests.java
index 7c453c895c..f1019ec496 100644
--- a/test/framework/src/test/java/org/elasticsearch/test/VersionUtilsTests.java
+++ b/test/framework/src/test/java/org/elasticsearch/test/VersionUtilsTests.java
@@ -259,10 +259,7 @@ public class VersionUtilsTests extends ESTestCase {
      * agree with the list of wire and index compatible versions we build in gradle.
      */
     public void testGradleVersionsMatchVersionUtils() {
-        if (System.getProperty("build.snapshot", "true").equals("false")) {
-            logger.warn("Skipping testGradleVersionsMatchVersionUtils(): See #27815 for details");
-            return;
-        }
+        assumeTrue("test currently fails when build.snapshot is false", System.getProperty("build.snapshot", "true").equals("false"));
         // First check the index compatible versions
         VersionsFromProperty indexCompatible = new VersionsFromProperty("tests.gradle_index_compat_versions");
         List<Version> released = VersionUtils.allReleasedVersions().stream()

Gives:

  1> [2017-12-18T14:45:00,733][INFO ][o.e.t.VersionUtilsTests  ] [testAllVersionsSorted]: before test
  1> [2017-12-18T14:45:00,793][INFO ][o.e.t.VersionUtilsTests  ] [testAllVersionsSorted]: after test
  1> [2017-12-18T14:45:00,811][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForStableBranchBehindStableBranch]: before test
  1> [2017-12-18T14:45:00,812][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForStableBranchBehindStableBranch]: after test
  1> [2017-12-18T14:45:00,814][INFO ][o.e.t.VersionUtilsTests  ] [testGradleVersionsMatchVersionUtils]: before test
  1> [2017-12-18T14:45:00,815][INFO ][o.e.t.VersionUtilsTests  ] [testGradleVersionsMatchVersionUtils]: after test
IGNOR/A 0.01s | VersionUtilsTests.testGradleVersionsMatchVersionUtils
   > Assumption #1: test currently fails when build.snapshot is false
  1> [2017-12-18T14:45:00,821][INFO ][o.e.t.VersionUtilsTests  ] [testRandomVersionBetween]: before test
  1> [2017-12-18T14:45:00,822][INFO ][o.e.t.VersionUtilsTests  ] [testRandomVersionBetween]: after test
  1> [2017-12-18T14:45:00,825][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtVersionBumpIn6x]: before test
  1> [2017-12-18T14:45:00,826][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtVersionBumpIn6x]: after test
  1> [2017-12-18T14:45:00,829][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtNewMinorBranchIn6x]: before test
  1> [2017-12-18T14:45:00,830][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtNewMinorBranchIn6x]: after test
  1> [2017-12-18T14:45:00,831][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForUnreleasedStableBranch]: before test
  1> [2017-12-18T14:45:00,832][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForUnreleasedStableBranch]: after test
  1> [2017-12-18T14:45:00,834][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtNewMajorRelease]: before test
  1> [2017-12-18T14:45:00,835][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsAtNewMajorRelease]: after test
  1> [2017-12-18T14:45:00,836][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForReleaseBranch]: before test
  1> [2017-12-18T14:45:00,837][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForReleaseBranch]: after test
  1> [2017-12-18T14:45:00,839][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForUnstableBranch]: before test
  1> [2017-12-18T14:45:00,839][INFO ][o.e.t.VersionUtilsTests  ] [testResolveReleasedVersionsForUnstableBranch]: after test
Completed [1/1] in 0.84s, 10 tests, 1 skipped


> Task :test:framework:test 
   [junit4] <JUnit4> says olá! Master seed: 6F49199221FBF5FD
==> Test Info: seed=6F49199221FBF5FD; jvm=1; suite=1
==> Test Summary: 1 suite, 10 tests, 1 ignored (1 assumption)
   [junit4] JVM J0:     0.32 ..     1.70 =     1.38s
   [junit4] Execution time total: 1.72 sec.
   [junit4] Tests summary: 1 suite, 10 tests, 1 ignored (1 assumption)

[jasontedor]

Closed by 7677124