StompBrokerRelayMessageHandler cannot reconnect to host after host relocation to new IP and dns change [SPR-13702]

[spring-projects-issues]

Juliusz Piwoński opened SPR-13702 and commented

Scenario 1: StompBrokerRelay is initiated correctly and connected to host e.g. foobar.com which was resolved to lets say 192.168.1.100.
Then we start a new server (Rabbit) on another IP address and re-point the DNS name foobar.com to the new address.
Unfortunatelly NettyTcpClient still tries to connect to the old address which is accompanied by the following log entry

2015-11-18 23:52:16.809  INFO 28858 --- [eactor-tcp-io-7] r.io.net.impl.netty.tcp.NettyTcpClient   : Failed to connect to foobar.com/192.168.1.100:61613. Attempting reconnect in 5000ms.

||
Scenario 2: similar scenario when resolving host name fails (DNS broken) during StompBrokerRelayMessageHandler initialization.
In consequence handler will never connect to host, even after DNS server is fixed and running.

I'm not sure whose responsibility is to resolve host name again (StompBrokerRelayMessageHandler, Reactor2TcpClient or Netty) but I would point to Reactor2TcpClient client as currently resolving dns names is done only once in:
reactor.io.net.Spec.TcpClientSpec called by Reactor2TcpClient constructor.

I think similar issue was mentioned in reactor/reactor#379
and one of the possible solution could be exposing ReconnectStrategy to configuration (as mentioned a the end of conversation: rstoyanchev/spring-websocket-portfolio#47).
Or simply make resolving host name always with each reconnect attempt (currently fixed 5s).

Is that correct or am I missing something?

environment:

org.springframework:spring-messaging:4.2.1.RELEASE
io.projectreactor:reactor-net:2.0.5.RELEASE
io.netty:netty-all:4.0.32.Final

---

Affects: 4.2.1

1 votes, 5 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

It looks like we can make an improvement in Reactor2TcpClient so that our ReactorReconnectAdapter re-resolves the socket address using the original host and port rather than relying on the one passed into it.

It also seems like a fix on the Reactor-side (i.e. issue #379) would be needed so that when a TcpClientSpec is built with a host and a port, it should re-resolve the address before passing it into Reconnect. This would be the only way when Reactor2TcpClient was created with a TcpClientFactory, in that case we wouldn't know the original host and port.

/cc smaldini

[spring-projects-issues]

Juergen Hoeller commented

Let's move this one to the 4.3 timeframe, in particular since it involves an update in several places. We may still decide to pull it forward to a 4.2.x release eventually, of course.

Juergen

[spring-projects-issues]

Rossen Stoyanchev commented

I left a comment under Reactor #379. I don't think we can address this on our side alone.

[spring-projects-issues]

Rossen Stoyanchev commented

Juliusz Piwoński it looks like this is going to work out to be a fix on the Reactor side (see this commit). The supplied host and port should now be re-resolved on each attempt to connect. This is currently available in Reactor 2.0.8 snapshots. Would you mine giving it a try to confirm it works for you?

[spring-projects-issues]

Juliusz Piwoński commented

Would you mine giving it a try to confirm it works for you?
Sure, I'll test this, just give me a while please.

[spring-projects-issues]

Rossen Stoyanchev commented

Sure and thanks.

[spring-projects-issues]

Juliusz Piwoński commented

Rossen Stoyanchev: I've tested both scenarios (wrong dns at startup and dynamic dns change while service is running) with libs:

compile 'org.springframework:spring-messaging:4.2.5.RELEASE'
compile 'io.projectreactor:reactor-net:2.0.8.BUILD-SNAPSHOT'
compile 'io.netty:netty-all:4.0.32.Final'

Looks like both cases WORK OK now :) Thanks!

Below a piece of log confirming that (initially name "krolik" points to my local address, then it is switched to rabbit broker inside docker container):

2016-03-15 13:14:29.330  INFO 12084 --- [eactor-tcp-io-8] r.io.net.impl.netty.tcp.NettyTcpClient   : CONNECTED: [id: 0x64b6e4db, /192.168.110.54:52884 => krolik/192.168.110.54:61613]
..
2016-03-15 13:15:20.926  INFO 12084 --- [eactor-tcp-io-8] r.io.net.impl.netty.tcp.NettyTcpClient   : CLOSED: [id: 0x64b6e4db, /192.168.110.54:52884 :> krolik/192.168.110.54:61613]
2016-03-15 13:15:20.926  INFO 12084 --- [eactor-tcp-io-8] r.io.net.impl.netty.tcp.NettyTcpClient   : Failed to connect to reactor.io.net.impl.netty.tcp.NettyTcpClient$ReconnectingChannelListener$3@3c66bcc2. Attempting reconnect in 5000ms.
...
2016-03-15 13:18:27.398  INFO 12084 --- [TaskExecutor-72] o.s.a.r.c.CachingConnectionFactory       : Created new connection: SimpleConnection@72f6c2c4 [delegate=amqp://guest@172.17.0.4:5672/]
2016-03-15 13:18:30.880  INFO 12084 --- [eactor-tcp-io-6] r.io.net.impl.netty.tcp.NettyTcpClient   : CONNECTED: [id: 0x15a17887, /172.17.0.1:51677 => krolik/172.17.0.4:61613]
2016-03-15 13:18:30.884  INFO 12084 --- [eactor-tcp-io-6] o.s.m.s.s.StompBrokerRelayMessageHandler : "System" session connected.
2016-03-15 13:18:30.885  INFO 12084 --- [eactor-tcp-io-6] o.s.m.s.s.StompBrokerRelayMessageHandler : BrokerAvailabilityEvent[available=true, StompBrokerRelay[krolik:61613]

[spring-projects-issues]

Nick Hristov commented

Is this issue going to make it's way into any release?

This scenario is anything but minor. In an environment such as AWS where ip addresses can change frequently this issue can become a real deployment blocker.

[spring-projects-issues]

Rossen Stoyanchev commented

Nick Hristov, as indicated in comments above, this issue was addressed with issue 379 in io.projectreactor:reactor-net version 2.0.8, and the fix was confirmed by others.

Note that Spring Framework 5 is based on io.projectreactor.ipc:reactor-netty, currently on 0.7.x (not the same as reactor-net). If you're having this issue in Spring Framework 5, and in any case, please open a new ticket with a fresh description.