[SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost #25803

gaborgsomogyi · 2019-09-16T12:14:48Z

What changes were proposed in this pull request?

KafkaDelegationTokenSuite fails on different platforms with the following problem:

19/09/11 11:07:42.690 pool-1-thread-1-SendThread(localhost:44965) DEBUG ZooKeeperSaslClient: creating sasl client: Client=zkclient/[email protected];service=zookeeper;serviceHostname=localhost.localdomain
...
NIOServerCxn.Factory:localhost/127.0.0.1:0: Zookeeper Server failed to create a SaslServer to interact with a client during session initiation:
javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
	at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:125)
	at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
	at javax.security.sasl.Sasl.createSaslServer(Sasl.java:524)
	at org.apache.zookeeper.util.SecurityUtils$2.run(SecurityUtils.java:233)
	at org.apache.zookeeper.util.SecurityUtils$2.run(SecurityUtils.java:229)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.zookeeper.util.SecurityUtils.createSaslServer(SecurityUtils.java:228)
	at org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:44)
	at org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:38)
	at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:186)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:227)
	at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
	at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
	at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
	at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
	at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62)
	at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
	at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
	... 13 more
NIOServerCxn.Factory:localhost/127.0.0.1:0: Client attempting to establish new session at /127.0.0.1:33742
SyncThread:0: Creating new log file: log.1
SyncThread:0: Established session 0x100003736ae0000 with negotiated timeout 10000 for client /127.0.0.1:33742
pool-1-thread-1-SendThread(localhost:35625): Session establishment complete on server localhost/127.0.0.1:35625, sessionid = 0x100003736ae0000, negotiated timeout = 10000
pool-1-thread-1-SendThread(localhost:35625): ClientCnxn:sendSaslPacket:length=0
pool-1-thread-1-SendThread(localhost:35625): saslClient.evaluateChallenge(len=0)
pool-1-thread-1-EventThread: zookeeper state changed (SyncConnected)
NioProcessor-1: No server entry found for kerberos principal name zookeeper/[email protected]
NioProcessor-1: No server entry found for kerberos principal name zookeeper/[email protected]
NioProcessor-1: Server not found in Kerberos database (7)
NioProcessor-1: Server not found in Kerberos database (7)

The problem reproducible if the localhost and localhost.localdomain order exhanged:

[systest@gsomogyi-build spark]$ cat /etc/hosts
127.0.0.1   localhost.localdomain localhost localhost4 localhost4.localdomain4
::1         localhost.localdomain localhost localhost6 localhost6.localdomain6

The main problem is that ZkClient connects to the canonical loopback address (which is not necessarily localhost).

Why are the changes needed?

KafkaDelegationTokenSuite failed in some environments.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit tests on different platforms.

…nical host name differs from localhost

gaborgsomogyi · 2019-09-16T12:15:46Z

@koertkuipers may I ask to test this PR on your environment?

HeartSaVioR · 2019-09-16T12:31:30Z

The code change looks good. Would like to see the result of CI build to ensure the change doesn't break existing ones. (CI build seems to be down right now.)

gaborgsomogyi · 2019-09-16T12:32:50Z

Yeah, my intention is to test 3 ways (apart from my local clusters):

mvn PR builder
sbt PR builder
@koertkuipers env

koertkuipers · 2019-09-16T14:01:08Z

ok i will test

SparkQA · 2019-09-16T19:46:04Z

Test build #110631 has finished for PR 25803 at commit 2d9b8df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-09-16T19:57:46Z

Kafka flakyness:
Caused by: sbt.ForkMain$ForkError: java.lang.AssertionError: assertion failed: Partition [topic-2, 0] metadata not propagated after timeout

gaborgsomogyi · 2019-09-16T19:57:55Z

retest this please

SparkQA · 2019-09-16T21:03:17Z

Test build #110664 has finished for PR 25803 at commit 2d9b8df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-09-16T21:12:26Z

Same failure: it might be related to the change. Once the build fails for same reason it seems to be high likely due to the change.

HeartSaVioR · 2019-09-16T21:12:32Z

retest this, please

SparkQA · 2019-09-16T22:02:29Z

Test build #110668 has finished for PR 25803 at commit 2d9b8df.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

koertkuipers · 2019-09-16T23:34:53Z

this fixed works for my environment. thanks!

HeartSaVioR · 2019-09-16T23:49:35Z

Glad to hear! If we run some more CI builds and verify they work without high flakiness, it should be good.

HeartSaVioR · 2019-09-16T23:49:43Z

retest this, please

SparkQA · 2019-09-17T00:52:47Z

Test build #110677 has finished for PR 25803 at commit 2d9b8df.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-09-17T00:54:29Z

retest this, please

SparkQA · 2019-09-17T01:41:34Z

Test build #110707 has finished for PR 25803 at commit 2d9b8df.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR

LGTM

gaborgsomogyi · 2019-09-17T07:32:40Z

Just to be on the safe side testing with maven as well

gaborgsomogyi · 2019-09-17T07:32:47Z

retest this please

SparkQA · 2019-09-17T08:13:14Z

Test build #110754 has finished for PR 25803 at commit 2d9b8df.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-09-17T08:15:04Z

So all the tests pass what I wanted. Thanks @koertkuipers and @HeartSaVioR

vanzin · 2019-09-17T22:29:57Z

Merging to master.

[SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback cano…

2d9b8df

…nical host name differs from localhost

dongjoon-hyun added the TESTS label Sep 16, 2019

HeartSaVioR approved these changes Sep 17, 2019

View reviewed changes

gaborgsomogyi changed the title ~~[SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost~~ [SPARK-29027][TESTS][maven] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost Sep 17, 2019

gaborgsomogyi changed the title ~~[SPARK-29027][TESTS][maven] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost~~ [SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost Sep 17, 2019

vanzin closed this in 71e7516 Sep 17, 2019

[SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost #25803

[SPARK-29027][TESTS] KafkaDelegationTokenSuite fix when loopback canonical host name differs from localhost #25803

Uh oh!

Conversation

gaborgsomogyi commented Sep 16, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gaborgsomogyi commented Sep 16, 2019

Uh oh!

HeartSaVioR commented Sep 16, 2019

Uh oh!

gaborgsomogyi commented Sep 16, 2019

Uh oh!

koertkuipers commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

gaborgsomogyi commented Sep 16, 2019

Uh oh!

gaborgsomogyi commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

HeartSaVioR commented Sep 16, 2019

Uh oh!

HeartSaVioR commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

koertkuipers commented Sep 16, 2019

Uh oh!

HeartSaVioR commented Sep 16, 2019

Uh oh!

HeartSaVioR commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

HeartSaVioR commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

HeartSaVioR left a comment

Choose a reason for hiding this comment

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants