Painless lambda tests are failing on JDK 9 build 158

[jasontedor]

Many of the Painless tests related to lambdas have started failing since upgrading the CI to JDK 9 build 158 with errors like:

ScriptException[runtime error]; nested: BootstrapMethodError[call site initialization exception]; nested: LambdaConversionException[Type mismatch for instantiated parameter 0: int is not a subtype of class java.lang.Object];
	at __randomizedtesting.SeedInfo.seed([E6F5130EFC315FC3:5D0BF4794861CE3E]:0)
	at org.elasticsearch.painless.ScriptImpl.convertToScriptException(ScriptImpl.java:181)
	at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:128)
	at org.elasticsearch.painless.ScriptTestCase.exec(ScriptTestCase.java:91)
	at org.elasticsearch.painless.ScriptTestCase.exec(ScriptTestCase.java:72)
	at org.elasticsearch.painless.ScriptTestCase.exec(ScriptTestCase.java:60)
	at org.elasticsearch.painless.LambdaTests.testPrimitiveLambdasWithTypedArgs(LambdaTests.java:56)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:547)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.BootstrapMethodError: call site initialization exception
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:384)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:250)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:240)
	at org.elasticsearch.painless.Executable$Script.execute(List l = new ArrayList(); l.add(1); l.add(1); return l.stream().mapToInt(int x -> x + 1).sum(); @ <inline script>:64)
	at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:123)
	... 40 more
Caused by: java.lang.invoke.LambdaConversionException: Type mismatch for instantiated parameter 0: int is not a subtype of class java.lang.Object
	at java.base/java.lang.invoke.AbstractValidatingLambdaMetafactory.checkDescriptor(AbstractValidatingLambdaMetafactory.java:308)
	at java.base/java.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:296)
	at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(LambdaMetafactory.java:473)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:343)
	... 44 more

[jasontedor]

I pushed commits to mute these tests on JDK 9:

master: 8e09eca
5.x: f437391
5.3: ec9c798

[jasontedor]

JDK 9 will enter rampdown phase 2 on 2017-03-16 (March 16, 2017); after that point, only show-stopping bugs will be fixed so we should try to understand the situation here as soon as possible in case there's an issue within JDK 9.

[rjernst]

We understand what is going on: java made lambda factory less lenient (http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/4be3ef759ead). @jdconrad is working on a fix to add additional bridge methods to compensate when necessary.

[uschindler]

Hi, I have to try out, but the problematic part seems to be here: https://github.com/elastic/elasticsearch/blob/master/modules/lang-painless/src/main/java/org/elasticsearch/painless/FunctionRef.java#L194-L196

This only "fixes" the issue for the return type, but not the parameter types. So it looks like we need to wrap/box also the parameter types. Also we have to take care of the void type!

There are two problems in the changes in Java 9 that may lead to this problem:

• The boxing does no longer work if it is Object.class, so the same trick as applied to return type should be applied also to parameters
• There is a special check for void.class in the new Java 9 code. I think if we have a return type of void we must manually adapt it to fix the issue.

I just have to get Gradle working with build 159...

[jdconrad]

@uschindler So, I may be on a different page here, but I believe when I tried the solution you're suggesting here it would no longer allow primitives to be widened, though the LambdaMetaFactory says this:

Reference ->	Primitive	
for parameter types: Q is a primitive wrapper and Primitive(Q) can be widened to S 
for return types: If Q is a primitive wrapper, check that Primitive(Q) can be widened to S

I think there is a bug here on there end for that. Maybe you could confirm this?

I also think we may need Object to be convertible to primitive in the same way it was prior to the changes, though.

For example, consider this:

        assertEquals(2, exec("List l = new ArrayList(); l.add((short)1); l.add(1); "
                           + "return l.stream().mapToLong(long x -> x).sum();"));

if we convert the Object -> Long via the LambdaMetaFactory, I think that breaks the def type for the List values since they can't be cast from Short -> Long and Integer -> Long. I think what we want is for LambdaMetaFactory to use asType for conversions exactly as MethodHandles do. That would also prevent illegal narrowing.

Also, thank you for taking a look! It's much appreciated.

[uschindler]

Hi,
in my testings with b159 yesterday, I can confirm, its not working as documented. So we should really open a bug! It would be horrible if this stays like this, as the whole Painless Lambdas stop working, unless we reimplement the whole stuff.

I think there is a bug here on there end for that. Maybe you could confirm this?

I think so. It also refuses to work to box/unbox under certain conditions. It was lkate yesterday, but we should write a bug report and add a test similar to the tests in the OpenJDK commit.

I am a bit afraid: The issue about this is "private", so I have the feeling there was a security issue reported to Oracle. If this is the case, we may expect that they backport it to Java 8 (we should try the preview release!!!). And then we have a desaster!

[uschindler]

In addition the Oracle patch is buggy, see: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/4be3ef759ead#l1.54

This line also prevents boxing/unboxing, and that is what I also discovered with my patch. It fails because int.class is not assignable to Integer.class and vice versa. So we have no chance to fix the whole thing at all. I discovered that in my tests yesterday.

[uschindler]

At least I can confirm: The preview build of Java 8 update 152 is not affected by this. But it is already one month old, so maybe it does not yet contain it!