(WIP) Support for tile-based geo grid aggregation #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ications (elastic#36249) This pull request adds the TransportShardCloseAction which is a transport replication action that acquires all index shard permits for its execution. This action will be used in the future by the MetaDataIndexStateService in a new index closing process, where we need to execute some sanity checks before closing an index. The action executes the following verifications on the primary and replicas: * there is no other on going operation active on the shard * the data node holding the shard knows that the index is blocked for writes * the shard's max sequence number is equal to the global checkpoint When the verifications are done and successful, the shard is flushed. Relates elastic#33888
The commit changes how indices are closed in the MetaDataIndexStateService. It now uses a 3 steps process where writes are blocked on indices to be closed, then some verifications are done on shards using the TransportVerifyShardBeforeCloseAction added in elastic#36249, and finally indices states are moved to CLOSE and their routing tables removed. The closing process also takes care of using the pre-7.0 way to close indices if the cluster contains mixed version of nodes and a node does not support the TransportVerifyShardBeforeCloseAction. It also closes unassigned indices. Related to elastic#33888
…ation (elastic#36755) This pull request modifies the TransportVerifyShardBeforeCloseAction so that it marks unavailable shards as stale.
…y Shard actions (elastic#36630) This pull request changes the Freeze Index and Close Index actions so that these actions always requires a Task. The task's id is then propagated from the Freeze action to the Close action, and then to the Verify shard action. This way it is possible to track which Freeze task initiates the closing of an index, and which consecutive verifiy shard are executed for the index closing.
This commit adds a unique id to cluster blocks, so that they can be uniquely identified if needed. This is important for the Close Index API where multiple concurrent closing requests can be executed at the same time. By adding a UUID to the cluster block, we can generate unique "closing block" that can later be verified on shards and then checked again from the cluster state before closing the index. When the verification on shard is done, the closing block is replaced by the regular INDEX_CLOSED_BLOCK instance. If something goes wrong, calling the Open Index API will remove the block. Related to elastic#33888
…26719) * [Analysis] Deprecate Standard Html Strip Analyzer Deprecate only Standard Html Strip Analyzer If user create index with the analyzer since 7.0, es throws an exception. If an index was created before 7.0, es issue deprecation log We will remove it in 8.0 Related elastic#4704
This bug was introduced in elastic#36893 and had the effect that execution would continue after calling onFailure on the the listener in checkIfTokenIsValid in the case that the token is expired. In a case of many consecutive requests this could lead to the unwelcome side effect of an expired access token producing a successful authentication response.
This commit adds a more user friendly error message when a license signature is malformed/truncated in a way that it cannot be meaningfully parsed.
We already had logic to stop datafeeds running against jobs that were OPENING, but a job that relocates from one node to another while OPENED stays OPENED, and this could cause the datafeed to fail when it sent data to the OPENED job on its new node before it had a corresponding autodetect process. This change extends the check to stop datafeeds running when their job is OPENING _or_ stale (i.e. has not had its status reset since relocating to a different node). Relates elastic#36810
…s.testConcurrentRefresh
The test intercepts TransportVerifyShardBeforeCloseAction shard requests, so it needs a minimum of 2 primary shards on 2 different nodes to correctly intercepts requests.
…etTotalHits This change turns an assertion into an IllegalStateException in SearchPhaseController#getTotalHits. The goal is to help identify the cause of the failures in elastic#37179 which seems to fail only in CI. The assertion will be restored when the issue is solved (NORELEASE).
This change adds support for the 'include_type_name' parameter for the indices.get API. This parameter, which defaults to `false` starting in 7.0, changes the response to not include the indices type names any longer. If the parameter is set in the request, we additionally emit a deprecation warning since using the parameter should be only temporarily necessary while adapting to the new response format and we will remove it with the next major version.
Tracked in elastic#36779
* TESTS: Real Coordinator in SnapshotServiceTests * Introduce real coordinator in SnapshotServiceTests to be able to test network disruptions realistically * Make adjustments to cluster applier service so that we can pass a mocked single threaded executor for tests
…aggregator (elastic#36748) Update BucketSortPipelineAggregator to use a List and Collections.sort() for sorting instead of a priority queue. This preserves the order for equal values. Closes elastic#36322.
Upgrading the Elastic Stack perfectly documents the process to upgrade ES from 5 to 6 when internal indices are present. However, the rolling upgrade docs do not mention anything about internal indices. This adds a warning in the rolling upgrade procedure, highlighting that internal indices should be upgraded before the rolling upgrade procedure can be started.
testing convetions need to be disabled if the test task is for fips.
Closing a channel using TLS/SSL requires reading and writing a CLOSE_NOTIFY message (for pre-1.3 TLS versions). Many implementations do not actually send the CLOSE_NOTIFY message, which means we are depending on the TCP close from the other side to ensure channels are closed. In case there is an issue with this, we need a timeout. This commit adds a timeout to the channel close process for TLS secured channels. As part of this change, we need a timer service. We could use the generic Elasticsearch timeout threadpool. However, it would be nice to have a local to the nio event loop timer service dedicated to network needs. In the future this service could support read timeouts, connect timeouts, request timeouts, etc. This commit adds a basic priority queue backed service. Since our timeout volume (channel closes) is very low, this should be fine. However, this can be updated to something more efficient in the future if needed (timer wheel). Everything being local to the event loop thread makes the logic simple as no locking or synchronization is necessary.
This commit moves log statements related to classification of naming convention checks for tests to debug level. At info level they emit an enormous amount of output in CI, while these are not generally useful for debugging normal build failures.
This commit makes the assemble tasks in the bwc projects noops by setting the dependsOn directly. While we can not remove things from dependsOn, we can still completely override the dependencies. closes elastic#33581
This commit fixes a race condition in a test introduced by elastic#36900 that verifies concurrent authentications get a result propagated from the first thread that attempts to authenticate. Previously, a thread may be in a state where it had not attempted to authenticate when the first thread that authenticates finishes the authentication, which would cause the test to fail as there would be an additional authentication attempt. This change adds additional latches to ensure all threads have attempted to authenticate before a result gets returned in the thread that is performing authentication.
* ML: add migrate anomalies assistant * adjusting failure handling for reindex * Fixing request and tests * Adding tests to blacklist * adjusting test * test fix: posting data directly to the job instead of relying on datafeed * adjusting API usage * adding Todos and adjusting endpoint * Adding types to reindexRequest * removing unreliable "live" data test * adding index refresh to test * adding index refresh to test * adding index refresh to yaml test * fixing bad exists call * removing todo * Addressing remove comments * Adjusting rest endpoint name * making service have its own logger * adjusting validity check for newindex names * fixing typos * fixing renaming
…ic#37207) * ingest: compile mustache template only if field includes '{{'' Prior to this change, any field in an ingest node processor that supports script templates would be compiled as mustache template regardless if they contain a template or not. Compiling normal text as mustache templates is harmless. However, each compilation counts against the script compilation circuit breaker. A large number of processors without any templates or scripts could un-intuitively trip the too many script compilations circuit breaker. This change simple checks for '{{' in the text before it attempts to compile. fixes elastic#37120
Collaborator
Author
|
closing in favor of this PR and its plans elastic#37596 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP new configurable geogrid implementation