Skip to content

Conversation

@ywelsch
Copy link
Contributor

@ywelsch ywelsch commented Dec 8, 2018

Moves all remaining (rolling-upgrade and mixed-version) REST tests to use Zen2. To avoid adding extra configuration, it relies on Zen2 being set as the default discovery type. This required a few smaller changes in other tests. I've removed AzureMinimumMasterNodesTests which tests Zen1 functionality and dates from a time where host providers were not configurable and each cloud plugin had its own discovery.type, subclassing the ZenDiscovery class. I've also adapted a few tests which were unnecessarily adding addTestZenDiscovery = false for the same legacy reasons. Finally, this also moves the unconfigured-node-name REST test to Zen2, testing the auto-bootstrapping functionality in development mode when no discovery configuration is provided.

@ywelsch ywelsch added >non-issue v7.0.0 :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. labels Dec 8, 2018
@ywelsch ywelsch requested a review from DaveCTurner December 8, 2018 12:14
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@DaveCTurner
Copy link
Contributor

Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good but I have questioned a couple of the test fixes.

@ywelsch
Copy link
Contributor Author

ywelsch commented Dec 8, 2018

packaging-sample failed because auto-bootstrapping failed with

[2018-12-08T13:52:49,579][WARN ][o.e.c.c.ClusterBootstrapService] [localhost.localdomain] discovery attempt failed
# org.elasticsearch.transport.Remote# org.elasticsearch.transport.Remote
TransportException: [localhost.localdomain][127.0.0.1:9300][cluster:monitor/discovered_nodes]
# Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/discovered_nodes]
TransportException: [localhost.localdomain][127.0.0.1:9300][cluster:monitor/discovered_nodes]
# Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/discovered_nodes]
# 	at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
# 	at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:152) ~[?:?]
# 	at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:107) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:524) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:346) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:351) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:263) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$7(AuthenticationService.java:234) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:252) ~[?:?]
# 	at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
# 	at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:152) ~[?:?]
# 	at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:107) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:524) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:346) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:351) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:263) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$7(AuthenticationService.java:234) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:252) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:186) ~[?:?]
# 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:304) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:182) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:186) ~[?:?]
# 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:304) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:182) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:213) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:178) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:137) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:107) ~[?:?]
# 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:119) ~[?:?]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:213) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:178) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:137) ~[?:?]
# 	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:107) ~[?:?]
# 	at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:119) ~[?:?]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
# 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:710) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:83) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:153) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:656) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:710) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.access$000(TransportService.java:83) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:153) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:656) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:156) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$400(SecurityServerTransportInterceptor.java:52) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:134) [x-pack-security-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:571) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:546) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.cluster.coordination.ClusterBootstrapService$1.run(ClusterBootstrapService.java:108) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:467) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
# 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191]
# 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
# 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_191]
# 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
# 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
# 	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:571) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:546) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.cluster.coordination.ClusterBootstrapService$1.run(ClusterBootstrapService.java:108) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:467) [elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
# 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
# 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191]
# 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
# 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_191]
# 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
# 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
# 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

and ci-2 failed because of #36380

For the first failure, I've tried making that action internal and see if that helps.

public class GetDiscoveredNodesAction extends Action<GetDiscoveredNodesResponse> {
public static final GetDiscoveredNodesAction INSTANCE = new GetDiscoveredNodesAction();
public static final String NAME = "cluster:monitor/discovered_nodes";
public static final String NAME = "internal:cluster/discovered_nodes";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't immediately see how this would help, since both are covered by system privileges:

"internal:*",
"indices:monitor/*", // added for monitoring
"cluster:monitor/*", // added for monitoring
"cluster:admin/bootstrap_cluster", // for the bootstrap service
"cluster:admin/reroute", // added for DiskThresholdDecider.DiskListener
"indices:admin/mapping/put", // needed for recovery and shrink api
"indices:admin/template/put", // needed for the TemplateUpgradeService
"indices:admin/template/delete", // needed for the TemplateUpgradeService
"indices:admin/seq_no/global_checkpoint_sync*", // needed for global checkpoint syncs
"indices:admin/settings/update" // needed for DiskThresholdMonitor.markIndicesReadOnly

Not that it won't help, just that I don't understand the mechanism.

DaveCTurner
DaveCTurner previously approved these changes Dec 8, 2018
@DaveCTurner DaveCTurner dismissed their stale review December 8, 2018 16:15

Review comments addressed, but need another look

@ywelsch
Copy link
Contributor Author

ywelsch commented Dec 8, 2018

@DaveCTurner I figured out the issue: The system context which we set in ClusterBootstrapService does not get preserved across the ThreadPool#schedule[UnlessShuttingDown] methods. The following patch fixed the issue. I don't want to make that change as part of this PR, because I'm not exactly sure about the implications just yet. For now I've opted to just fix the problem for this specific situation here (see 1f1a0d3), but will follow-up with another PR for the schedule problem.

diff --git a/server/src/main/java/org/elasticsearch/threadpool/ThreadPool.java b/server/src/main/java/org/elasticsearch/threadpool/ThreadPool.java
index bae801492e1..a55b992ad00 100644
--- a/server/src/main/java/org/elasticsearch/threadpool/ThreadPool.java
+++ b/server/src/main/java/org/elasticsearch/threadpool/ThreadPool.java
@@ -349,7 +349,8 @@ public class ThreadPool implements Scheduler, Closeable {
         if (!Names.SAME.equals(executor)) {
             command = new ThreadedRunnable(command, executor(executor));
         }
-        return scheduler.schedule(new ThreadPool.LoggingRunnable(command), delay.millis(), TimeUnit.MILLISECONDS);
+        return scheduler.schedule(new ThreadPool.LoggingRunnable(
+            threadContext.preserveContext(command)), delay.millis(), TimeUnit.MILLISECONDS);
     }
 
     public void scheduleUnlessShuttingDown(TimeValue delay, String executor, Runnable command) {

@DaveCTurner
Copy link
Contributor

Good catch. Does a similar issue exist in DiscoveryUpgradeService, which also schedules things that look like they need their context preserving? Not sure how the X-pack rolling-upgrade tests haven't caught that if so.

@ywelsch
Copy link
Contributor Author

ywelsch commented Dec 8, 2018

Just after a quick look. it seems to me that DiscoveryUpgradeService is only triggering internal actions, which don't exhibit this issue (see AuthorizationService.shouldReplaceUserWithSystem which calls isInternalAction)

@ywelsch ywelsch requested a review from DaveCTurner December 9, 2018 11:40
Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. >non-issue v7.0.0-beta1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants