SSH Daemon (NIO2) is not starting anymore

[josivan]

Hello

I have two VM running gitblit under jetty version 9.2.11. Both are using java 8 (lastest update "1.8.0_77") in a ubuntu server 14.04.

The only difference that I've noticed is the kernel version

One of them is
2.6.32-042stab111.12 #1 SMP Thu Sep 17 11:38:20 MSK 2015 x86_64 x86_64 x86_64 GNU/Linux (lets call server 1)
and the other one is
_2.6.32-042stab113.21 #1 SMP Wed Mar 23 11:05:25 MSK 2016 x86_64 x86_64 x86_64 GNU/Linux _(lets call server 2)

On server 1, everything works fine.

But on Server 2, the context of gitblit is not up and runinng.

The last record in log is:

2016-04-12 22:22:53 [INFO ] Federation passphrase is blank! This server can not be PULLED from.
2016-04-12 22:22:53 [INFO ] Fanout PubSub service is disabled.
2016-04-12 22:22:53 [INFO ] Git Daemon is listening on 0.0.0.0:9419

After that, jetty's service is failing, and context isn't available. The application stays with status STARTING forever.

I've tried to reinstall ssh server and client with no success.

Can someone help me with that?

Regards

[josivan]

A few days after I post my question I've found the correct answer.

I followed the steps bellow to detect the root cause of problem:

1. I've downloaded the source code of gitblit's version I'm using. In this case, version 1.7.1 available here.
2. With source code I added some quick logs (using sysout) only to check when the application freezing. I noticed that problem was in code of Apache MINA sshd.
3. I've also downloaded the source code of Apache MINA sshd to debug. In this case I wasn't able to add sysout, then I chose to do a remote debugging in my jetty running on server as described here.

Of course that it was a little bit slowly, but I noticed that when the sshd's code called SecureRandom.generateSeed(8) in class SecurityUtils.BouncyCastleRandom

public BouncyCastleRandom() {
    ValidateUtils.checkTrue(isBouncyCastleRegistered(), "BouncyCastle not registered");
    this.random = new VMPCRandomGenerator();
    byte[] seed = new SecureRandom().generateSeed(8);
    this.random.addSeedMaterial(seed);
}

The system used to freeze completely.

After a long time searching on the internet I've found this link/blog https://blog.cloudflare.com/ensuring-randomness-with-linuxs-random-number-generator/ and when I typed the command cat /proc/sys/kernel/random/entropy_avail I was always getting 0 (zero) as result.

I was aware that my Linux is a VPS running under openvz hosted in host1plus. Based on that, I asked to VPS provider check the reason that in my VPS I was always getting zero as result.

The answer from Technical support was:

We have enabled the random device for your VPS. Please check if it works works for you and if the issue is solved.

After that update my gitblit was backing to up and running state.

[gitblit]

Wow. Very interesting details. Thank you for sharing!