Add support for Hazelcast 4

[justinnichols]

With Hazelcast 4.0, Spring Boot 2.2.6 is not properly adhering to the spring.hazelcast.config application property which allows pointing to a specific Hazelcast configuration and then instantiating the proper bean.

Spring Boot is attempting to create a HazelcastInstance of HazelcastClientConfiguration (which expects <hazelcast-client> XML root tag) instead of HazelcastServerConfiguration (which expects <hazelcast> XML root tag).

The error that occurs:

Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.

The following Github repository depicts two example Spring Boot 2.2.6 projects, one with this error, and one with a workaround:

https://github.com/justinnichols/spring-boot-2.2.6-hazelcast-4.0

Dependencies:

<dependency>
	<groupId>com.hazelcast</groupId>
	<artifactId>hazelcast</artifactId>
	<version>4.0</version>
</dependency>

Configuration in application.properties:

spring.hazelcast.config=classpath:config/hazelcast.xml

The stacktrace that occurs is:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [org/springframework/boot/autoconfigure/hazelcast/HazelcastClientConfiguration$HazelcastClientConfigFileConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1290) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1210) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	... 18 common frames omitted

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
	... 32 common frames omitted

Caused by: com.hazelcast.config.InvalidConfigurationException: Invalid root element in xml configuration! Expected: <hazelcast-client>, Actual: <hazelcast>.
	at com.hazelcast.client.config.XmlClientConfigBuilder.checkRootElement(XmlClientConfigBuilder.java:183) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.parseAndBuildConfig(XmlClientConfigBuilder.java:168) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:157) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:150) ~[hazelcast-4.0.jar:4.0]
	at com.hazelcast.client.config.XmlClientConfigBuilder.build(XmlClientConfigBuilder.java:145) ~[hazelcast-4.0.jar:4.0]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientFactory.getClientConfig(HazelcastClientFactory.java:66) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientFactory.<init>(HazelcastClientFactory.java:48) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientConfiguration$HazelcastClientConfigFileConfiguration.hazelcastInstance(HazelcastClientConfiguration.java:55) ~[spring-boot-autoconfigure-2.2.6.RELEASE.jar:2.2.6.RELEASE]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_192]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_192]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_192]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_192]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]

Workarounds (kindly provided by @mesutcelik via Gitter)

• Place the hazelcast.xml in the root of the classpath. (This is depicted in the repo linked above)
• Specify a hazelcast.config system property.
• Define a configuration bean programmatically.

[wilkinsona]

Thanks for the report. We currently build and test against Hazelcast 3.12.6. I'm not surprised to learn that the auto-configuration does not work out of the box with Hazelcast 4.0 as it's a new major version.

[vpavic]

Note that besides a handful of binary changes made without deprecation period, Hazelcast 4.0 also made changes to packaging and started including client components into the main artifact (see hazelcast/hazelcast#7448). IMO this will make supporting both 3.6 and 4.0 quite challenging from Boot's perspective.

[justinnichols]

The following also works, but it definitely assumes much more than the generic auto-configuration. It will work in the meantime for our application:

import com.hazelcast.config.Config;
import com.hazelcast.config.XmlConfigBuilder;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;

import java.io.IOException;

@Configuration
public class HazelcastConfiguration {
    @Autowired ResourceLoader resourceLoader;

    @Value("${spring.hazelcast.config:classpath:hazelcast.xml}")
    private String hazelcastConfig;

    @Bean
    HazelcastInstance hazelcastInstance() throws IOException  {
        Resource configAsResource = resourceLoader.getResource(hazelcastConfig);

        Config config = new XmlConfigBuilder(configAsResource.getInputStream()).build();
        return Hazelcast.newHazelcastInstance(config);
    }
}

[snicoll]

This is also blocked on micrometer-metrics/micrometer#1697 and if Micrometer drops support for Hazelcast 3.x, that would put pressure on us to upgrade for 2.3 indeed.

The change Vedran mentioned (thanks @vpavic!) is indeed very annoying as it breaks our checks to determine whether the user wants an embedded server or connect to an existing one with the client. We'll probably have to introduce a property with an enum and the user will have to opt-in for the behaviour they want.

[snicoll]

I've added a comment on hazelcast/hazelcast#7448 (comment). If Hazelcast 4.1 split jars after all, then the enum I've just mentioned would be useless. I don't think we should upgrade until this is clarified.

[snicoll]

The current problem that has been described here is due to the merging that Vedran described above. Now that we have both the embedded server and the client on the classpath, the auto-configuration for the client always matches altough the configuration points to an embedded server. You'd get the same problem if you put hazelcast-client on your classpath altough you want to use an embedded server so there might be something we can do here with the enum.

I am going to give that a try and see if the health indicator and cache infrastructure still works. Metrics won't but that's currently worked on the micrometer side of things.

Given that Hazelcast has dropped support for hazelcast-client, this is also a breaking change from a dependency management perspective and you won't be able to simply override the hazelcast version with hazelcast.version unless we start tracking the client with a different property.

[snicoll]

The health indicator is also broken

Caused by: java.lang.NoSuchMethodError: com.hazelcast.core.HazelcastInstance.getLocalEndpoint()Lcom/hazelcast/core/Endpoint;
	at org.springframework.boot.actuate.hazelcast.HazelcastHealthIndicator.lambda$doHealthCheck$0(HazelcastHealthIndicator.java:47) ~[spring-boot-actuator-2.3.0.BUILD-20200407.071319-511.jar:2.3.0.BUILD-SNAPSHOT]
	at com.hazelcast.transaction.impl.TransactionManagerServiceImpl.executeTransaction(TransactionManagerServiceImpl.java:119) ~[hazelcast-4.0.jar:4.0]
	... 47 common frames omitted

We can "fix" that with an additional class check so that it backs off with Hazelcast 4.