Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions docs/reference/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ include::setup.asciidoc[]

include::setup/setup-xes.asciidoc[]

include::monitoring/configuring-monitoring.asciidoc[]

include::setup/setup-xclient.asciidoc[]

include::setup/bootstrap-checks-xes.asciidoc[]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
[role="xpack"]
[testenv="gold"]
[[collecting-monitoring-data]]
=== Collecting monitoring data
++++
<titleabbrev>Collecting monitoring data</titleabbrev>
++++
== Collecting monitoring data

If you enable the Elastic {monitor-features} in your cluster, you can
optionally collect metrics about {es}. By default, monitoring is enabled but
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/monitoring/collectors.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ For more information about the configuration options for the collectors, see

[float]
[[es-monitoring-stack]]
=== Collecting data from across the Elastic Stack
==== Collecting data from across the Elastic Stack

{monitoring} in {es} also receives monitoring data from other parts of the
Elastic Stack. In this way, it serves as an unscheduled monitoring data
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/monitoring/configuring-metricbeat.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="gold"]
[[configuring-metricbeat]]
=== Collecting {es} monitoring data with {metricbeat}
== Collecting {es} monitoring data with {metricbeat}

[subs="attributes"]
++++
Expand Down
20 changes: 0 additions & 20 deletions docs/reference/monitoring/configuring-monitoring.asciidoc

This file was deleted.

3 changes: 0 additions & 3 deletions docs/reference/monitoring/exporters.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,3 @@ which is used to determine whether the resource should be replaced. The `version
field value represents the latest version of {monitoring} that changed the
resource. If a resource is edited by someone or something external to
{monitoring}, those changes are lost the next time an automatic update occurs.

include::local-export.asciidoc[]
include::http-export.asciidoc[]
40 changes: 40 additions & 0 deletions docs/reference/monitoring/how-monitoring-works.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[role="xpack"]
[testenv="basic"]
[[how-monitoring-works]]
== How monitoring works
++++
<titleabbrev>How it works</titleabbrev>
++++

Each {es} node, {ls} node, {kib} instance, and Beat is considered unique in the
cluster based on its persistent UUID, which is written to the
<<path-settings,`path.data`>> directory when the node or instance starts.

Monitoring documents are just ordinary JSON documents built by monitoring each
{stack} component at some collection interval. If you want to alter the
templates for these indices, see <<config-monitoring-indices>>.

Each component in the stack is responsible for monitoring itself and then
forwarding those documents to the {es} production cluster for both routing and
indexing (storage). The routing and indexing processes in {es} are handled by
what are called <<es-monitoring-collectors,collectors>> and
<<es-monitoring-exporters,exporters>>.

Alternatively, in 6.4 and later, you can use {metricbeat} to collect
monitoring data about {kib} and ship it directly to the monitoring cluster,
rather than routing it through the production cluster. In 6.5 and later, you
can also use {metricbeat} to collect and ship data about {es}.

To learn how to collect monitoring data, see:

* <<collecting-monitoring-data>>
* <<configuring-metricbeat>>
* {kibana-ref}/xpack-monitoring.html[Monitoring {kib}]
* {logstash-ref}/monitoring-logstash.html[Monitoring Logstash]
* Monitoring Beats:
** {auditbeat-ref}/monitoring.html[Auditbeat]
** {filebeat-ref}/monitoring.html[Filebeat]
** {heartbeat-ref}/monitoring.html[Heartbeat]
** {metricbeat-ref}/monitoring.html[Metricbeat]
** {packetbeat-ref}/monitoring.html[Packetbeat]
** {winlogbeat-ref}/monitoring.html[Winlogbeat]
2 changes: 1 addition & 1 deletion docs/reference/monitoring/http-export.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[http-exporter]]
=== HTTP Exporters
=== HTTP exporters

The `http` exporter is the preferred exporter in {monitoring} because it enables
the use of a separate monitoring cluster. As a secondary benefit, it avoids
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 20 additions & 44 deletions docs/reference/monitoring/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,54 +1,30 @@
[role="xpack"]
[testenv="basic"]
[[es-monitoring]]
= Monitoring {es}
[[monitor-elasticsearch-cluster]]
= Monitor a cluster

[partintro]
--
The Elastic {monitor-features} enable you to easily monitor the health of
your {es} cluster. The monitoring metrics are collected from each node and
stored in {es} indices.

TIP: In production environments, it is recommended to store the monitoring data
in a separate _monitoring cluster_. See
{stack-ov}/monitoring-production.html[Monitoring in a production environment].

Each {es} node is considered unique based on its persistent UUID, which is
written on first start to its <<path-settings,`path.data`>> directory, which
defaults to `./data`.

All settings associated with monitoring in {es} must be set in either the
`elasticsearch.yml` file for each node or, where possible, in the dynamic
cluster settings. For more information, see <<configuring-monitoring>>.

[[es-monitoring-overview]]
{es} is also at the core of monitoring across the {stack}. In all cases,
monitoring documents are just ordinary JSON documents built by monitoring each
{stack} component at some collection interval, then indexing those
documents into the monitoring cluster.

Each component in the stack is responsible for monitoring itself and then
forwarding those documents to the {es} production cluster for both routing and
indexing (storage). The routing and indexing processes in {es} are handled by
what are called <<es-monitoring-collectors,collectors>> and
<<es-monitoring-exporters,exporters>>.

Alternatively, in 6.4 and later, you can use {metricbeat} to collect
monitoring data about {kib} and ship it directly to the monitoring cluster,
rather than routing it through the production cluster. In 6.5 and later, you
can also use {metricbeat} to collect and ship data about {es}.

You can view monitoring data from {kib} where it’s easy to spot issues at a
glance or delve into the system behavior over time to diagnose operational
issues. In addition to the built-in status warnings, you can also set up custom
alerts based on the data in the monitoring indices.

For an introduction to monitoring your {stack}, including Beats, {ls}, and {kib},
see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

The {stack} {monitor-features} provide a way to keep a pulse on the health and
performance of your {es} cluster.

* <<monitoring-overview>>
* <<how-monitoring-works>>
* <<collecting-monitoring-data>>
* <<configuring-metricbeat>>
* <<config-monitoring-indices>>
* <<es-monitoring-collectors>>
* <<es-monitoring-exporters>>
--

include::overview.asciidoc[]
include::how-monitoring-works.asciidoc[]
include::collecting-monitoring-data.asciidoc[]
include::pause-export.asciidoc[]
include::configuring-metricbeat.asciidoc[]
include::indices.asciidoc[]
include::collectors.asciidoc[]
include::exporters.asciidoc[]
include::pause-export.asciidoc[]
include::local-export.asciidoc[]
include::http-export.asciidoc[]

2 changes: 1 addition & 1 deletion docs/reference/monitoring/indices.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[config-monitoring-indices]]
=== Configuring indices for monitoring
== Configuring indices for monitoring

<<indices-templates,Index templates>> are used to configure the indices
that store the monitoring data collected from a cluster.
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/monitoring/local-export.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[local-exporter]]
=== Local Exporters
=== Local exporters

The `local` exporter is the default exporter in {monitoring}. It routes data
back into the same (local) cluster. In other words, it uses the production
Expand Down Expand Up @@ -56,7 +56,7 @@ For more information about the configuration options for the `local` exporter,
see <<local-exporter-settings>>.

[[local-exporter-cleaner]]
==== Cleaner Service
==== Cleaner service

One feature of the `local` exporter, which is not present in the `http` exporter,
is a cleaner service. The cleaner service runs once per day at 01:00 AM UTC on
Expand Down
40 changes: 40 additions & 0 deletions docs/reference/monitoring/overview.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[role="xpack"]
[testenv="basic"]
[[monitoring-overview]]
== Monitoring overview
++++
<titleabbrev>Overview</titleabbrev>
++++

When you monitor a cluster, you collect data from the {es} nodes, {ls} nodes,
and {kib} instances in your cluster.

All of the monitoring metrics are stored in {es}, which enables you to easily
visualize the data from {kib}. By default, the monitoring metrics are stored in
local indices.

TIP: In production, we strongly recommend using a separate monitoring cluster. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents
monitoring activities from impacting the performance of your production cluster. For the same reason, we also recommend using a separate {kib} instance for
viewing the monitoring data.

The following diagram illustrates a typical monitoring architecture with separate
production and monitoring clusters:

image::monitoring/images/architecture10.png["A typical monitoring environment"]

In 6.4 and later, you can use {metricbeat} to collect and ship data about
{kib}, rather than routing it through {es}. In 6.5 and later, you can also use
{metricbeat} to collect and ship data about {es}. For example:

image::monitoring/images/architecture20.png[A typical monitoring environment that includes {metricbeat}]

If you have the appropriate license, you can route data from multiple production
clusters to a single monitoring cluster. For more information about the
differences between various subscription levels, see:
https://www.elastic.co/subscriptions

IMPORTANT: In general, the monitoring cluster and the clusters being monitored
should be running the same version of the stack. A monitoring cluster cannot
monitor production clusters running newer versions of the stack. If necessary,
the monitoring cluster can monitor production clusters running the latest
release of the previous major version.
2 changes: 1 addition & 1 deletion docs/reference/monitoring/pause-export.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[pause-export]]
== Pausing Data Collection
=== Pausing data collection

To stop generating {monitoring} data in {es}, disable data collection:

Expand Down
12 changes: 11 additions & 1 deletion docs/reference/redirects.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -649,4 +649,14 @@ See <<getting-started-next-steps>>.

[role="exclude",id="ccs-reduction"]
=== {ccs-cap} reduction
See <<ccs-works>>.
See <<ccs-works>>.

[role="exclude",id="es-monitoring"]
=== Monitoring {es}

See <<monitoring-overview>>.

[role="exclude",id="configuring-monitoring"]
=== Configuring monitoring in {es}

See <<monitor-elasticsearch-cluster>>.