Cross Cluster Replication support #3533

Mpdreamz · 2019-01-07T16:04:46Z

This adds support for all the CCR related API's

ccr.follow.json
ccr.follow_stats.json
ccr.get_auto_follow_pattern.json
ccr.pause_follow.json
ccr.put_auto_follow_pattern.json
ccr.resume_follow.json
ccr.stats.json
ccr.unfollow.json

This also introduces CoordinatedIntegrationTestBase<> as a means to execute integration tests in a forced sequence for all api overloads.

Further more it introduces RemoteClusterConfiguration to simplify specifying remote clusters as part of the cluster settings API for use with Cross Cluster replication and searches.

You can now also enable soft deletes on index settings in a strongly typed fashion.

russcam

I've left some comments

russcam · 2019-01-25T00:43:58Z

src/Nest/Cluster/ClusterSettings/ClusterPutSettings/RemoteClusterConfiguration.cs

+namespace Nest
+{
+	/// <summary>
+	/// This is a custom dictionary that helps in the creation of remote cluster configuration


Maybe just

Configuration used to configure a remote cluster

?

russcam · 2019-01-25T00:45:42Z

...CrossClusterReplication/AutoFollow/CreateAutoFollowPattern/CreateAutoFollowPatternRequest.cs

+namespace Nest
+{
+	/// <summary>
+	/// This API creates a new named collection of auto-follow patterns against the remote cluster specified


Drop the "This API" from the beginning

russcam · 2019-01-25T00:46:13Z

...usterReplication/AutoFollow/CreateAutoFollowPattern/ElasticClient-CreateAutoFollowPattern.cs

+	public partial interface IElasticClient
+	{
+		/// <summary>
+		/// This API creates a new named collection of auto-follow patterns against the remote cluster specified


Drop "This API" from beginning

russcam · 2019-01-25T00:47:04Z

...rossClusterReplication/AutoFollow/GetAutoFollowPattern/ElasticClient-GetAutoFollowPattern.cs

+{
+	public partial interface IElasticClient
+	{
+		/// <summary> This API gets configured auto-follow patterns. This API will return the specified auto-follow pattern collection. </summary>


Drop "This API" from beginning. Change "This API will return..." to "Returns..."

russcam · 2019-01-25T00:47:59Z

...XPack/CrossClusterReplication/AutoFollow/GetAutoFollowPattern/GetAutoFollowPatternRequest.cs

+
+namespace Nest
+{
+	/// <summary> This API gets configured auto-follow patterns. This API will return the specified auto-follow pattern collection. </summary>


Drop "This API" from beginning. Change "This API will return..." to "Returns..."

russcam · 2019-01-25T00:52:53Z

src/Nest/XPack/CrossClusterReplication/Follow/UnfollowIndex/UnfollowIndexRequest.cs

+namespace Nest
+{
+	/// <summary>
+	/// This API stops the following task associated with a follower index and removes index metadata and settings associated with


Drop "This API" from the beginning

russcam · 2019-01-25T00:53:01Z

src/Nest/XPack/CrossClusterReplication/Stats/CcrStatsRequest.cs

+namespace Nest
+{
+	/// <summary>
+	/// This API gets cross-cluster replication stats. This API will return all stats related to cross-cluster replication.


Drop "This API" from the beginning

russcam · 2019-01-25T00:53:12Z

src/Nest/XPack/CrossClusterReplication/Stats/ElasticClient-CcrStats.cs

+	public partial interface IElasticClient
+	{
+		/// <summary>
+		/// This API gets cross-cluster replication stats. This API will return all stats related to cross-cluster replication.


Drop "This API" from the beginning

russcam · 2019-01-25T02:01:25Z

src/Tests/Tests/XPack/CrossClusterReplication/CrossClusterReplicationFollowTests.cs

+namespace Tests.XPack.CrossClusterReplication
+{
+	[SkipVersion("<6.5.0", "")]
+	public class CrossClusterReplicationFollowTests : CoordinatedIntegrationTestBase<WritableCluster>


There's a lot going on in this integration test! I think it may be useful to add a <summary> that provides a brief overview of what is being tested.

As I understand it, each step is dependent on the "setup" of the previous one, and is asserted in its own integration test method in the class.

If my understanding is correct, maybe we could do something similar here as to what exists for ML job API integration tests - a base class with setup steps that can be called in IntegrationSetup method for each deriving integration test, and a derived test class for each separate API?

russcam · 2019-01-25T02:16:06Z

src/Tests/Tests/XPack/CrossClusterReplication/CrossClusterReplicationAutoFollowTests .cs

+namespace Tests.XPack.CrossClusterReplication
+{
+	[SkipVersion("<6.5.0", "")]
+	public class CrossClusterReplicationAutoFollowTests : CoordinatedIntegrationTestBase<WritableCluster>


Similar to other comment:

As I understand it, each step is dependent on the "setup" of the previous one, and is asserted in its own integration test method in the class.

If my understanding is correct, maybe we could do something similar here as to what exists for ML job API integration tests - a base class with setup steps that can be called in IntegrationSetup method for each deriving integration test, and a derived test class for each separate API?

Aye, The ML approach is valid since most operate on a job so their IntegrationSetup is relatively straightforward and inheritable.

The setup in ML run constantly though for each ApiTest test class.

CCR is more a chain of API's to call, where the same approach would leave a lot of unfinished setup chains e.g CoodinatedIntegrationTest allows us to express this dependency chain:

A1 => A2 => A3 => A4 = > A5 => A6

Which will be only executed once for each permutation of [async, sync] and [fluent, ois]

if we take the object graph approach to test each AN in isolation

We end up with:

A1 A1 => A2 A1 => A2 => A3 A1 => A2 => A3 => A4 A1 => A2 => A3 => A4 = > A5 A1 => A2 => A3 => A4 = > A5 => A6

Foreach permutation of [async, sync] and [fluent, ois].

…ods)

… for it

Mpdreamz · 2019-01-25T11:04:08Z

@russcam thank you for the review, I've updated the xmldocs to be less This API heavy 😸

CoordinatedIntegrationTestBase is an approach I definitely want to keep for complicated family of API's that do not exist in isolation. E.g ccr autofollow but also application privileges (still working on that in a branch)) and the ILM API's from 6.6 will benefit from this less rigid approach to integration tests.

* Initial code gen for CCR * Created all the infra for the CCR api's (request/response/client methods) * Progress commit implementing the API and coordinated integration test for it * Add global ccr stats endpoint and tidy up implemented ccr api's thus far * Auto Follow CCR api's + tests * default keyword * CCR url and api tests * This API comments

* Initial code gen for CCR * Created all the infra for the CCR api's (request/response/client methods) * Progress commit implementing the API and coordinated integration test for it * Add global ccr stats endpoint and tidy up implemented ccr api's thus far * Auto Follow CCR api's + tests * default keyword * CCR url and api tests * This API comments (cherry picked from commit 13c6a80)

Mpdreamz added Feature labels Jan 7, 2019

Mpdreamz force-pushed the v65/ccr branch from 003abb7 to fd6ac62 Compare January 24, 2019 13:13

russcam self-assigned this Jan 25, 2019

russcam reviewed Jan 25, 2019

View reviewed changes

Mpdreamz added 8 commits January 25, 2019 12:00

Initial code gen for CCR

bcaf020

Created all the infra for the CCR api's (request/response/client meth…

5c26b7b

…ods)

Progress commit implementing the API and coordinated integration test…

fa97046

… for it

Add global ccr stats endpoint and tidy up implemented ccr api's thus far

05ce639

Auto Follow CCR api's + tests

d1fb692

default keyword

e8023b9

CCR url and api tests

4609dc7

This API comments

fcda832

Mpdreamz force-pushed the v65/ccr branch from fd6ac62 to fcda832 Compare January 25, 2019 11:00

Mpdreamz merged commit 3e60dab into 6.5 Jan 25, 2019

Mpdreamz deleted the v65/ccr branch January 25, 2019 11:04

Cross Cluster Replication support #3533

Cross Cluster Replication support #3533

Uh oh!

Conversation

Mpdreamz commented Jan 7, 2019

Uh oh!

russcam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mpdreamz commented Jan 25, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants