SearchGroupsResolverInMemoryTests leaking threads

[gwbrown]

Build scan: sample scan

Repro line: None given

Reproduces locally?: n/a, see above

Applicable branches: 8.0, master

Failure history: build-stats - looks like failures with this error go back to Oct. 18 at least.

Failure excerpt:

	com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.elasticsearch.xpack.security.authc.ldap.SearchGroupsResolverInMemoryTests: 	
	   1) Thread[id=124, name=Timer thread for LDAPConnection(name='test-connection-testSearchTimeoutIsFailure', not connected), state=WAITING, group=TGRP-SearchGroupsResolverInMemoryTests]	
	        at java.base@17.0.1/java.lang.Object.wait(Native Method)	
	        at java.base@17.0.1/java.lang.Object.wait(Object.java:338)	
	        at java.base@17.0.1/java.util.TimerThread.mainLoop(Timer.java:537)	
	        at java.base@17.0.1/java.util.TimerThread.run(Timer.java:516)	
		at __randomizedtesting.SeedInfo.seed([42DA9C1586F63FD5]:0)	

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[davidkyle]

Another failure https://gradle-enterprise.elastic.co/s/egyt6poyjpmcc/tests/:x-pack:plugin:security:test/org.elasticsearch.xpack.security.authc.ldap.SearchGroupsResolverInMemoryTests/classMethod?top-execution=1

[tvernum]

This one has me stumped so far.
I cannot get it to fail locally despite thousands of executions. I don't know what to make of that.

I suspect it's a real issue (though maybe only in test code), but I cannot make sense of it.

The Timer thread is blocked on Object.wait
That object should get a notify when the timer is cancelled, which should happen when the connection is closed, which should happen @After the test completes.

I cannot work out which of those steps is failing, and since I can't reproduce it, I'm having trouble getting any diagnostic on it.
I suspect the next step will be to try and reproduce on a CI worker. If I can get it failing there I can possibly put in a patched version of the LDAP SDK to try and diagnose where the cancel is breaking down.

[tvernum]

Update: I successfully reproduced the error after 100 attempts on a Debian 10 CI Worker.
I'm now retrying with a patched version of the test and the LDAP SDK that will give more detail when it fails. However, after more than 600 attempts I have not had a failure with the patched version 😿

[tvernum]

Update: After 3000 executions it failed.

It appears to be some form of locking contention (possibly a deadlock, but I haven't gotten deep enough to know).
When we try to close the connection, it attempts to obtain a lock on itself before it clears the Timer. It seems to be blocking on synchronized(this), so it can never clear the timer. (it wasn't)

I've added more logging and will keep trying.

[henningandersen]

Had a PR build fail with this leak error today.

[pugnascotia]

Another failure: https://gradle-enterprise.elastic.co/s/3wjqo4lrylpqc

[joegallo]

Me too: https://gradle-enterprise.elastic.co/s/4c3dojkii4vai/

[tvernum]

I think I've tracked it down and raised an upstream issue pingidentity/ldapsdk#120 (it looks like a real, though very infrequent thread leak in LDAP SDK).

I'll see if there's something we can do locally to avoid this until we can get a fix from upstream.