-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Introduce CCR getting started guide #35434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
cd2d9d8
61373e9
296ae89
83f020e
60237df
d6b8a4a
0244e9c
c8512da
a93f973
a265806
39720b2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,329 @@ | ||
| [role="xpack"] | ||
| [testenv="platinum"] | ||
| [[ccr-getting-started]] | ||
| == Getting Started | ||
| == Getting Started with {ccr} | ||
|
|
||
| beta[] | ||
| This is the getting started section of the {ccr} docs. | ||
|
|
||
| This getting-started guide for {ccr} shows you how to: | ||
|
|
||
| * <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote | ||
| cluster>> | ||
| * <<ccr-getting-started-leader-index,Create a leader index>> in a remote cluster | ||
| * <<ccr-getting-started-follower-index,Create a follower index>> that replicates | ||
| a leader index | ||
| * <<ccr-getting-started-auto-follow,Automatically create follower indices>> | ||
|
|
||
| [float] | ||
| [[ccr-getting-started-before-you-begin]] | ||
| === Before you begin | ||
| . {stack-gs}/get-started-elastic-stack.html#install-elasticsearch[Install {es}] | ||
| on your local and remote clusters. | ||
|
|
||
| . Obtain a license that includes the {ccr} features. See | ||
| https://www.elastic.co/subscriptions[subscriptions] and | ||
| <<license-management>>. | ||
|
|
||
| . If the Elastic {security-features} are enabled in your local and remote | ||
| clusters, you need a user that has appropriate authority to perform the steps | ||
| in this tutorial. | ||
| + | ||
| -- | ||
| [[ccr-getting-started-security]] | ||
| The {ccr} features use cluster privileges and built-in roles to make it easier | ||
| to control which users have authority to manage {ccr}. | ||
|
|
||
| By default, you can perform all of the steps in this tutorial by | ||
| using the built-in `elastic` user. However, a password must be set for this user | ||
| before the user can do anything. For information about how to set that password, | ||
| see <<security-getting-started>>. | ||
|
|
||
| If you are performing these steps in a production environment, take extra care | ||
| because the `elastic` user has the `superuser` role and you could inadvertently | ||
| make significant changes. | ||
|
|
||
| Alternatively, you can assign the appropriate privileges to a user ID of your | ||
| choice. On the remote cluster that contains the leader index, a user will need | ||
| the `read_ccr` cluster privilege and `monitor` and `read` privileges on the | ||
| leader index. | ||
|
|
||
| [source,yml] | ||
| -------------------------------------------------- | ||
| ccr_user: | ||
| cluster: | ||
| - read_ccr | ||
| indices: | ||
| - names: [ 'leader-index' ] | ||
| privileges: | ||
| - monitor | ||
| - read | ||
| -------------------------------------------------- | ||
|
|
||
| On the local cluster that contains the follower index, the same user will need | ||
| the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and | ||
| `manage_follow_index` privileges on the follower index. | ||
|
|
||
| [source,yml] | ||
| -------------------------------------------------- | ||
| ccr_user: | ||
| cluster: | ||
| - manage_ccr | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see this manage_ccr privilege documented yet either: https://www.elastic.co/guide/en/elastic-stack-overview/master/security-privileges.html
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, we should add that there. |
||
| indices: | ||
| - names: [ 'follower-index' ] | ||
| privileges: | ||
| - monitor | ||
| - read | ||
| - write | ||
| - manage_follow_index | ||
| -------------------------------------------------- | ||
|
|
||
| If you are managing | ||
| <<ccr-getting-started-remote-cluster,connecting to the remote cluster>> via the | ||
| cluster update settings API, you will also need a user with the `all` cluster | ||
| privilege. | ||
| -- | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think you'll also need manage cluster privileges to change the cluster settings. |
||
|
|
||
| [float] | ||
| [[ccr-getting-started-remote-cluster]] | ||
| === Connecting to a remote cluster | ||
|
|
||
| The {ccr} features require that you | ||
| {ref}/modules-remote-clusters.html[connect your local cluster to a remote | ||
| cluster]. In this tutorial, we will connect our local cluster to a remote | ||
| cluster with the cluster alias `leader`. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /_cluster/settings | ||
| { | ||
| "persistent" : { | ||
| "cluster" : { | ||
| "remote" : { | ||
| "leader" : { | ||
| "seeds" : [ | ||
| "127.0.0.1:9300" <1> | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[setup:host] | ||
| // TEST[s/127.0.0.1:9300/\${transport_host}/] | ||
| <1> Specifies the hostname and transport port of a seed node in the remote | ||
| cluster. | ||
|
|
||
| You can verify that the local cluster is successfully connected to the remote | ||
| cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| GET /_remote/info | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| The API will respond by showing that the local cluster is connected to the | ||
| remote cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "leader" : { | ||
| "seeds" : [ | ||
| "127.0.0.1:9300" | ||
| ], | ||
| "connected" : true, <1> | ||
| "num_nodes_connected" : 1, <2> | ||
| "max_connections_per_cluster" : 3, | ||
| "initial_connect_timeout" : "30s", | ||
| "skip_unavailable" : false | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
| // TEST[s/127.0.0.1:9300/$body.leader.seeds.0/] | ||
| // TEST[s/"connected" : true/"connected" : $body.leader.connected/] | ||
| // TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/] | ||
| <1> This shows the local cluster is connected to the remote cluster with cluster | ||
| alias `leader` | ||
| <2> This shows the number of nodes in the remote cluster the local cluster is | ||
| connected to. | ||
|
|
||
| [float] | ||
| [[ccr-getting-started-leader-index]] | ||
| === Creating a leader index | ||
|
|
||
| Leader indices require special index settings to ensure that the operations that | ||
| need to be replicated are available when the | ||
| follower requests them from the leader. These settings are used to enable soft | ||
| deletes on the leader index and to control how many soft deletes are retained. A | ||
| _soft delete_ occurs whenever a document is deleted or updated. Soft deletes can | ||
| be enabled only on new indices created on or after {es} 6.5.0. | ||
|
|
||
| In the following example, we will create a leader index in the remote cluster: | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /server-metrics | ||
| { | ||
| "settings" : { | ||
| "index" : { | ||
| "number_of_shards" : 1, | ||
| "number_of_replicas" : 0, | ||
| "soft_deletes" : { | ||
| "enabled" : true, <1> | ||
| "retention" : { | ||
| "operations" : 1024 <2> | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "mappings" : { | ||
| "metric" : { | ||
| "properties" : { | ||
| "@timestamp" : { | ||
| "type" : "date" | ||
| }, | ||
| "accept" : { | ||
| "type" : "long" | ||
| }, | ||
| "deny" : { | ||
| "type" : "long" | ||
| }, | ||
| "host" : { | ||
| "type" : "keyword" | ||
| }, | ||
| "response" : { | ||
| "type" : "float" | ||
| }, | ||
| "service" : { | ||
| "type" : "keyword" | ||
| }, | ||
| "total" : { | ||
| "type" : "long" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
| <1> Enables soft deletes on the leader index. | ||
| <2> Sets that up to 1024 soft deletes will be retained. | ||
|
|
||
| [float] | ||
| [[ccr-getting-started-follower-index]] | ||
| === Creating a follower index | ||
|
|
||
| Follower indices are created with the {ref}/ccr-put-follow.html[create follower | ||
| API]. When you create a follower index, you must reference the | ||
| <<ccr-getting-started-remote-cluster,remote cluster>> and the | ||
| <<ccr-getting-started-leader-index,leader index>> that you created in the remote | ||
| cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /server-metrics-copy/_ccr/follow | ||
| { | ||
| "remote_cluster" : "leader", | ||
| "leader_index" : "server-metrics" | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "follow_index_created" : true, | ||
| "follow_index_shards_acked" : true, | ||
| "index_following_started" : true | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| Now when you index documents into your leader index, you will see these | ||
| documents replicated in the follower index. You can | ||
| inspect the status of replication using the | ||
| {ref}/ccr-get-follow-stats[get follower stats API]. | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| POST /server-metrics-copy/_ccr/pause_follow | ||
|
|
||
| POST /server-metrics-copy/_close | ||
|
|
||
| POST /server-metrics-copy/_ccr/unfollow | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [float] | ||
| [[ccr-getting-started-auto-follow]] | ||
| === Automatically create follower indices | ||
|
|
||
| The auto-follow feature in {ccr} helps for time series use cases where you want | ||
| to follow new indices that are periodically created in the remote cluster | ||
| (such as daily Beats indices). Auto-following is configured using the | ||
| {ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API]. With an | ||
| auto-follow pattern, you reference the | ||
| <<ccr-getting-started-remote-cluster,remote cluster>> that you connected your | ||
| local cluster to. You must also specify a collection of patterns that match the | ||
| indices you want to automatically follow. | ||
|
|
||
| For example: | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /_ccr/auto_follow/beats | ||
| { | ||
| "remote_cluster" : "leader", | ||
| "leader_index_patterns" : | ||
| [ | ||
| "metricbeat-*", <1> | ||
| "packetbeat-*" <2> | ||
| ], | ||
| "follow_index_pattern" : "{{leader_index}}-copy" <3> | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
| <1> Automatically follow new {metricbeat} indices. | ||
| <2> Automatically follow new {packetbeat} indices. | ||
| <3> The name of the follower index is derived from the name of the leader index | ||
| by adding the suffix `-copy` to the name of the leader index. | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "acknowledged" : true | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| DELETE /_ccr/auto_follow/beats | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this privilege be added to the list here?: https://www.elastic.co/guide/en/elastic-stack-overview/master/security-privileges.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.