Skip to content

Commit d71aa16

Browse files
jasontedorjasontedor
committed
Merge branch 'master' into replica-sequence-number-recovery
* master: (131 commits) Replace EngineClosedException with AlreadyClosedExcpetion (#22631) Remove HttpServer and HttpServerAdapter in favor of a simple dispatch method (#22636) move ignore parameter support from yaml test client to low level rest client (#22637) Fix thread safety of Stempel's token filter factory (#22610) Update profile.asciidoc Wrap rest httpclient with doPrivileged blocks (#22603) Revert "Add a deprecation notice to shadow replicas (#22025)" Revert "Don'y use `INDEX_SHARED_FS_ALLOW_RECOVERY_ON_ANY_NODE_SETTING` directly as it triggers (many) deprecation logging" Don'y use `INDEX_SHARED_FS_ALLOW_RECOVERY_ON_ANY_NODE_SETTING` directly as it triggers (many) deprecation logging Add a deprecation notice to shadow replicas (#22025) [Docs] Fix section title in profile.asciidoc ProfileResult and CollectorResult should print machine readable timing information (#22561) Add replica ops with version conflict to translog Replace custom Functional interface in ElasticsearchException with CheckedFunction Make RestChannelConsumer extend CheckedConsumer<RestChannel, Exception> replace ShardSearchRequest.FilterParser functional interface with CheckedFunction replace custom functional interface with CheckedFunction in percolate module [TEST] replace SizeFunction with Function<Integer, Integer> Expose CheckedFunction Expose logs base path ...
2 parents 8960522 + d80e3ee commit d71aa16

File tree

660 files changed

+9028
-13471
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

660 files changed

+9028
-13471
lines changed

‎TESTING.asciidoc‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,6 @@ run it using Gradle:
2525
gradle run
2626
-------------------------------------
2727

28-
or to attach a remote debugger, run it as:
29-
30-
-------------------------------------
31-
gradle run --debug-jvm
32-
-------------------------------------
33-
3428
=== Test case filtering.
3529

3630
- `tests.class` is a class-filtering shell-like glob pattern,
@@ -430,15 +424,33 @@ cd $BATS_ARCHIVES
430424
sudo -E bats $BATS_TESTS/*.bats
431425
-------------------------------------------------
432426

433-
Note: Starting vagrant VM outside of the elasticsearch folder requires to
434-
indicates the folder that contains the Vagrantfile using the VAGRANT_CWD
435-
environment variable:
427+
You can also use Gradle to prepare the test environment and then starts a single VM:
436428

437429
-------------------------------------------------
438-
gradle vagrantSetUp
439-
VAGRANT_CWD=/path/to/elasticsearch vagrant up centos-7 --provider virtualbox
430+
gradle vagrantFedora24#up
440431
-------------------------------------------------
441432

433+
Or any of vagrantCentos6#up, vagrantDebian8#up, vagrantFedora24#up, vagrantOel6#up,
434+
vagrantOel7#up, vagrantOpensuse13#up, vagrantSles12#up, vagrantUbuntu1204#up,
435+
vagrantUbuntu1604#up.
436+
437+
Once up, you can then connect to the VM using SSH from the elasticsearch directory:
438+
439+
-------------------------------------------------
440+
vagrant ssh fedora-24
441+
-------------------------------------------------
442+
443+
Or from another directory:
444+
445+
-------------------------------------------------
446+
VAGRANT_CWD=/path/to/elasticsearch vagrant ssh fedora-24
447+
-------------------------------------------------
448+
449+
Note: Starting vagrant VM outside of the elasticsearch folder requires to
450+
indicates the folder that contains the Vagrantfile using the VAGRANT_CWD
451+
environment variable.
452+
453+
442454
== Coverage analysis
443455

444456
Tests can be run instrumented with jacoco to produce a coverage report in
@@ -462,7 +474,7 @@ Combined (Unit+Integration) coverage:
462474
mvn -Dtests.coverage verify jacoco:report
463475
---------------------------------------------------------------------------
464476

465-
== Debugging from an IDE
477+
== Launching and debugging from an IDE
466478

467479
If you want to run elasticsearch from your IDE, the `gradle run` task
468480
supports a remote debugging option:
@@ -471,6 +483,17 @@ supports a remote debugging option:
471483
gradle run --debug-jvm
472484
---------------------------------------------------------------------------
473485

486+
== Debugging remotely from an IDE
487+
488+
If you want to run Elasticsearch and be able to remotely attach the process
489+
for debugging purposes from your IDE, can start Elasticsearch using `ES_JAVA_OPTS`:
490+
491+
---------------------------------------------------------------------------
492+
ES_JAVA_OPTS="-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=y" ./bin/elasticsearch
493+
---------------------------------------------------------------------------
494+
495+
Read your IDE documentation for how to attach a debugger to a JVM process.
496+
474497
== Building with extra plugins
475498
Additional plugins may be built alongside elasticsearch, where their
476499
dependency on elasticsearch will be substituted with the local elasticsearch

‎buildSrc/src/main/groovy/org/elasticsearch/gradle/test/RestTestPlugin.groovy‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ public class RestTestPlugin implements Plugin<Project> {
3939
if (false == REQUIRED_PLUGINS.any {project.pluginManager.hasPlugin(it)}) {
4040
throw new InvalidUserDataException('elasticsearch.rest-test '
4141
+ 'requires either elasticsearch.build or '
42-
+ 'elasticsearch.standalone-test')
42+
+ 'elasticsearch.standalone-rest-test')
4343
}
4444

4545
RestIntegTestTask integTest = project.tasks.create('integTest', RestIntegTestTask.class)

‎buildSrc/src/main/groovy/org/elasticsearch/gradle/test/StandaloneRestTestPlugin.groovy‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,9 @@ public class StandaloneRestTestPlugin implements Plugin<Project> {
4040
@Override
4141
public void apply(Project project) {
4242
if (project.pluginManager.hasPlugin('elasticsearch.build')) {
43-
throw new InvalidUserDataException('elasticsearch.standalone-test, '
44-
+ 'elasticsearch.standalone-test, and elasticsearch.build are '
45-
+ 'mutually exclusive')
43+
throw new InvalidUserDataException('elasticsearch.standalone-test '
44+
+ 'elasticsearch.standalone-rest-test, and elasticsearch.build '
45+
+ 'are mutually exclusive')
4646
}
4747
project.pluginManager.apply(JavaBasePlugin)
4848
project.pluginManager.apply(RandomizedTestingPlugin)

‎buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -404,10 +404,6 @@ class VagrantTestPlugin implements Plugin<Project> {
404404
args 'halt', box
405405
}
406406
stop.dependsOn(halt)
407-
if (project.extensions.esvagrant.boxes.contains(box) == false) {
408-
// we only need a halt task if this box was not specified
409-
continue;
410-
}
411407

412408
Task update = project.tasks.create("vagrant${boxTask}#update", VagrantCommandTask) {
413409
boxName box
@@ -435,6 +431,11 @@ class VagrantTestPlugin implements Plugin<Project> {
435431
dependsOn update
436432
}
437433

434+
if (project.extensions.esvagrant.boxes.contains(box) == false) {
435+
// we d'ont need tests tasks if this box was not specified
436+
continue;
437+
}
438+
438439
Task smoke = project.tasks.create("vagrant${boxTask}#smoketest", Exec) {
439440
environment vagrantEnvVars
440441
dependsOn up

‎buildSrc/src/main/resources/checkstyle_suppressions.xml‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,6 @@
311311
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]AlreadyExpiredException.java" checks="LineLength" />
312312
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]CompositeIndexEventListener.java" checks="LineLength" />
313313
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]IndexSettings.java" checks="LineLength" />
314-
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]IndexingSlowLog.java" checks="LineLength" />
315314
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]MergePolicyConfig.java" checks="LineLength" />
316315
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]SearchSlowLog.java" checks="LineLength" />
317316
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]analysis[/\\]AnalysisRegistry.java" checks="LineLength" />
@@ -430,7 +429,6 @@
430429
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]monitor[/\\]jvm[/\\]GcNames.java" checks="LineLength" />
431430
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]monitor[/\\]jvm[/\\]HotThreads.java" checks="LineLength" />
432431
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]node[/\\]Node.java" checks="LineLength" />
433-
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]node[/\\]internal[/\\]InternalSettingsPreparer.java" checks="LineLength" />
434432
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]plugins[/\\]PluginsService.java" checks="LineLength" />
435433
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]repositories[/\\]RepositoriesService.java" checks="LineLength" />
436434
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]repositories[/\\]Repository.java" checks="LineLength" />
@@ -442,7 +440,6 @@
442440
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]repositories[/\\]fs[/\\]FsRepository.java" checks="LineLength" />
443441
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]repositories[/\\]uri[/\\]URLIndexShardRepository.java" checks="LineLength" />
444442
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]repositories[/\\]uri[/\\]URLRepository.java" checks="LineLength" />
445-
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]rest[/\\]BytesRestResponse.java" checks="LineLength" />
446443
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]rest[/\\]RestController.java" checks="LineLength" />
447444
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]rest[/\\]action[/\\]cat[/\\]RestCountAction.java" checks="LineLength" />
448445
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]rest[/\\]action[/\\]cat[/\\]RestIndicesAction.java" checks="LineLength" />
@@ -467,7 +464,6 @@
467464
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]filters[/\\]InternalFilters.java" checks="LineLength" />
468465
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]geogrid[/\\]GeoHashGridAggregator.java" checks="LineLength" />
469466
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]global[/\\]GlobalAggregator.java" checks="LineLength" />
470-
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]global[/\\]InternalGlobal.java" checks="LineLength" />
471467
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]histogram[/\\]HistogramAggregator.java" checks="LineLength" />
472468
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]missing[/\\]InternalMissing.java" checks="LineLength" />
473469
<suppress files="core[/\\]src[/\\]main[/\\]java[/\\]org[/\\]elasticsearch[/\\]search[/\\]aggregations[/\\]bucket[/\\]missing[/\\]MissingAggregator.java" checks="LineLength" />
@@ -788,7 +784,6 @@
788784
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]search[/\\]nested[/\\]LongNestedSortingTests.java" checks="LineLength" />
789785
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]search[/\\]nested[/\\]NestedSortingTests.java" checks="LineLength" />
790786
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]shard[/\\]IndexShardTests.java" checks="LineLength" />
791-
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]shard[/\\]IndexingOperationListenerTests.java" checks="LineLength" />
792787
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]shard[/\\]ShardPathTests.java" checks="LineLength" />
793788
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]similarity[/\\]SimilarityTests.java" checks="LineLength" />
794789
<suppress files="core[/\\]src[/\\]test[/\\]java[/\\]org[/\\]elasticsearch[/\\]index[/\\]snapshots[/\\]blobstore[/\\]FileInfoTests.java" checks="LineLength" />

‎buildSrc/version.properties‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ lucene = 6.4.0-snapshot-084f7a0
44
# optional dependencies
55
spatial4j = 0.6
66
jts = 1.13
7-
jackson = 2.8.1
7+
jackson = 2.8.6
88
snakeyaml = 1.15
99
# When updating log4j, please update also docs/java-api/index.asciidoc
1010
log4j = 2.7

‎client/rest/src/main/java/org/elasticsearch/client/RestClient.java‎

Lines changed: 39 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
import java.util.Collection;
5050
import java.util.Collections;
5151
import java.util.Comparator;
52+
import java.util.HashMap;
5253
import java.util.HashSet;
5354
import java.util.Iterator;
5455
import java.util.List;
@@ -282,15 +283,44 @@ public void performRequestAsync(String method, String endpoint, Map<String, Stri
282283
public void performRequestAsync(String method, String endpoint, Map<String, String> params,
283284
HttpEntity entity, HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
284285
ResponseListener responseListener, Header... headers) {
285-
URI uri = buildUri(pathPrefix, endpoint, params);
286+
Objects.requireNonNull(params, "params must not be null");
287+
Map<String, String> requestParams = new HashMap<>(params);
288+
//ignore is a special parameter supported by the clients, shouldn't be sent to es
289+
String ignoreString = requestParams.remove("ignore");
290+
Set<Integer> ignoreErrorCodes;
291+
if (ignoreString == null) {
292+
if (HttpHead.METHOD_NAME.equals(method)) {
293+
//404 never causes error if returned for a HEAD request
294+
ignoreErrorCodes = Collections.singleton(404);
295+
} else {
296+
ignoreErrorCodes = Collections.emptySet();
297+
}
298+
} else {
299+
String[] ignoresArray = ignoreString.split(",");
300+
ignoreErrorCodes = new HashSet<>();
301+
if (HttpHead.METHOD_NAME.equals(method)) {
302+
//404 never causes error if returned for a HEAD request
303+
ignoreErrorCodes.add(404);
304+
}
305+
for (String ignoreCode : ignoresArray) {
306+
try {
307+
ignoreErrorCodes.add(Integer.valueOf(ignoreCode));
308+
} catch (NumberFormatException e) {
309+
throw new IllegalArgumentException("ignore value should be a number, found [" + ignoreString + "] instead", e);
310+
}
311+
}
312+
}
313+
URI uri = buildUri(pathPrefix, endpoint, requestParams);
286314
HttpRequestBase request = createHttpRequest(method, uri, entity);
287315
setHeaders(request, headers);
288316
FailureTrackingResponseListener failureTrackingResponseListener = new FailureTrackingResponseListener(responseListener);
289317
long startTime = System.nanoTime();
290-
performRequestAsync(startTime, nextHost().iterator(), request, httpAsyncResponseConsumerFactory, failureTrackingResponseListener);
318+
performRequestAsync(startTime, nextHost().iterator(), request, ignoreErrorCodes, httpAsyncResponseConsumerFactory,
319+
failureTrackingResponseListener);
291320
}
292321

293322
private void performRequestAsync(final long startTime, final Iterator<HttpHost> hosts, final HttpRequestBase request,
323+
final Set<Integer> ignoreErrorCodes,
294324
final HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
295325
final FailureTrackingResponseListener listener) {
296326
final HttpHost host = hosts.next();
@@ -304,15 +334,15 @@ public void completed(HttpResponse httpResponse) {
304334
RequestLogger.logResponse(logger, request, host, httpResponse);
305335
int statusCode = httpResponse.getStatusLine().getStatusCode();
306336
Response response = new Response(request.getRequestLine(), host, httpResponse);
307-
if (isSuccessfulResponse(request.getMethod(), statusCode)) {
337+
if (isSuccessfulResponse(statusCode) || ignoreErrorCodes.contains(response.getStatusLine().getStatusCode())) {
308338
onResponse(host);
309339
listener.onSuccess(response);
310340
} else {
311341
ResponseException responseException = new ResponseException(response);
312342
if (isRetryStatus(statusCode)) {
313343
//mark host dead and retry against next one
314344
onFailure(host);
315-
retryIfPossible(responseException, hosts, request);
345+
retryIfPossible(responseException);
316346
} else {
317347
//mark host alive and don't retry, as the error should be a request problem
318348
onResponse(host);
@@ -329,13 +359,13 @@ public void failed(Exception failure) {
329359
try {
330360
RequestLogger.logFailedRequest(logger, request, host, failure);
331361
onFailure(host);
332-
retryIfPossible(failure, hosts, request);
362+
retryIfPossible(failure);
333363
} catch(Exception e) {
334364
listener.onDefinitiveFailure(e);
335365
}
336366
}
337367

338-
private void retryIfPossible(Exception exception, Iterator<HttpHost> hosts, HttpRequestBase request) {
368+
private void retryIfPossible(Exception exception) {
339369
if (hosts.hasNext()) {
340370
//in case we are retrying, check whether maxRetryTimeout has been reached
341371
long timeElapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime);
@@ -347,7 +377,7 @@ private void retryIfPossible(Exception exception, Iterator<HttpHost> hosts, Http
347377
} else {
348378
listener.trackFailure(exception);
349379
request.reset();
350-
performRequestAsync(startTime, hosts, request, httpAsyncResponseConsumerFactory, listener);
380+
performRequestAsync(startTime, hosts, request, ignoreErrorCodes, httpAsyncResponseConsumerFactory, listener);
351381
}
352382
} else {
353383
listener.onDefinitiveFailure(exception);
@@ -452,8 +482,8 @@ public void close() throws IOException {
452482
client.close();
453483
}
454484

455-
private static boolean isSuccessfulResponse(String method, int statusCode) {
456-
return statusCode < 300 || (HttpHead.METHOD_NAME.equals(method) && statusCode == 404);
485+
private static boolean isSuccessfulResponse(int statusCode) {
486+
return statusCode < 300;
457487
}
458488

459489
private static boolean isRetryStatus(int statusCode) {
@@ -510,7 +540,6 @@ private static HttpRequestBase addRequestBody(HttpRequestBase httpRequest, HttpE
510540
}
511541

512542
private static URI buildUri(String pathPrefix, String path, Map<String, String> params) {
513-
Objects.requireNonNull(params, "params must not be null");
514543
Objects.requireNonNull(path, "path must not be null");
515544
try {
516545
String fullPath;

‎client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@
2828
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
2929
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
3030

31+
import java.security.AccessController;
32+
import java.security.PrivilegedAction;
3133
import java.util.Objects;
3234

3335
/**
@@ -177,7 +179,12 @@ public RestClient build() {
177179
if (failureListener == null) {
178180
failureListener = new RestClient.FailureListener();
179181
}
180-
CloseableHttpAsyncClient httpClient = createHttpClient();
182+
CloseableHttpAsyncClient httpClient = AccessController.doPrivileged(new PrivilegedAction<CloseableHttpAsyncClient>() {
183+
@Override
184+
public CloseableHttpAsyncClient run() {
185+
return createHttpClient();
186+
}
187+
});
181188
RestClient restClient = new RestClient(httpClient, maxRetryTimeout, defaultHeaders, hosts, pathPrefix, failureListener);
182189
httpClient.start();
183190
return restClient;

0 commit comments

Comments
 (0)