Influxdb3 monitor metrics

.github/instructions/content.instructions.md

@@ -208,6 +208,23 @@ When building shared content, use the `show-in` and `hide-in` shortcodes to show
 or hide blocks of content based on the current InfluxDB product/version.
 For more information, see [show-in](#show-in) and [hide-in](#hide-in).
 
+#### Links in shared content
+
+When creating links in shared content files, use `/influxdb3/version/` instead of the `{{% product-key %}}` shortcode.
+The keyword `version` gets replaced during the build process with the appropriate product version.
+
+**Use this in shared content:**
+```markdown
+[Configuration options](/influxdb3/version/reference/config-options/)
+[CLI serve command](/influxdb3/version/reference/cli/influxdb3/serve/)
+```
+
+**Not this:**
+```markdown
+[Configuration options](/influxdb3/{{% product-key %}}/reference/config-options/)
+[CLI serve command](/influxdb3/{{% product-key %}}/reference/cli/influxdb3/serve/)
+```
+
 #### Shortcodes in Markdown files
 
 For the complete shortcodes reference, see `/.github/instructions/shortcodes-reference.instructions.md`.
@@ -219,4 +236,4 @@ For the complete shortcodes reference, see `/.github/instructions/shortcodes-ref
 - Format code examples to fit within 80 characters
 - Use long options in command line examples (`--option` instead of `-o`)
 - Use GitHub callout syntax for notes and warnings
-- Image naming: `project/version-context-description.png`
+- Image naming: `project/version-context-description.png`

TESTING.md

@@ -121,6 +121,117 @@ Potential causes:
   # This is ignored
   ```
 
+### Metrics Endpoint Testing
+
+The metrics testing suite validates InfluxDB 3 Core and Enterprise metrics in two phases:
+
+1. **Phase 1: Direct metrics validation** - Validates metric format, existence, and types by directly querying InfluxDB endpoints
+2. **Phase 2: Prometheus integration** - Validates Prometheus configuration, scraping, and relabeling work as documented
+
+#### Phase 1: Direct Metrics Validation
+
+The `test/influxdb3/metrics_endpoint_test.py` suite validates that InfluxDB 3 metrics endpoints expose all documented metrics in correct Prometheus format.
+
+**Basic Usage:**
+
+```bash
+# Using the wrapper script (recommended)
+./test/run-metrics-tests.sh
+
+# Direct execution with Docker Compose
+docker compose run --rm influxdb3-core-pytest test/influxdb3/metrics_endpoint_test.py
+
+# Run specific test
+docker compose run --rm influxdb3-core-pytest test/influxdb3/metrics_endpoint_test.py -k test_http_grpc_metrics
+```
+
+**Verbose Output:**
+
+Set `VERBOSE_METRICS_TEST=true` to see detailed output showing which metrics are searched and the actual matching lines from the Prometheus endpoint:
+
+```bash
+# With wrapper script
+VERBOSE_METRICS_TEST=true ./test/run-metrics-tests.sh
+
+# With Docker Compose
+VERBOSE_METRICS_TEST=true docker compose run --rm \
+  -e VERBOSE_METRICS_TEST \
+  influxdb3-core-pytest \
+  test/influxdb3/metrics_endpoint_test.py
+```
+
+Example verbose output:
+```
+TEST: HTTP/gRPC Metrics
+================================================================================
+
+✓ Searching for: http_requests_total
+  Found 12 total occurrences
+  Matches:
+    # HELP http_requests_total accumulated total requests
+    # TYPE http_requests_total counter
+    http_requests_total{method="GET",path="/metrics",status="aborted"} 0
+```
+
+#### Phase 2: Prometheus Integration Testing
+
+The `test/influxdb3/prometheus_integration_test.py` suite validates that Prometheus can scrape InfluxDB metrics and that the documented relabeling configuration works correctly.
+
+**What it validates:**
+- Prometheus service discovers InfluxDB targets
+- Scrape configuration works with authentication
+- Relabeling adds `node_name` and `node_role` labels correctly
+- Regex patterns in relabel_configs match documentation
+- PromQL queries using relabeled metrics work
+- Example queries from documentation execute successfully
+
+**Basic Usage:**
+
+```bash
+# Using the wrapper script (recommended)
+./test/run-metrics-tests.sh --prometheus
+
+# Run both direct and Prometheus tests
+./test/run-metrics-tests.sh --all
+
+# Direct execution
+./test/influxdb3/run-prometheus-tests.sh
+
+# With verbose output
+VERBOSE_PROMETHEUS_TEST=true ./test/influxdb3/run-prometheus-tests.sh
+```
+
+**What happens during Prometheus tests:**
+
+1. Starts Prometheus service with documented configuration from `test/influxdb3/prometheus.yml`
+2. Waits for Prometheus to discover and scrape both InfluxDB instances
+3. Validates relabeling adds `node_name` label (extracted from `__address__`)
+4. Validates relabeling adds `node_role` label (based on node name pattern)
+5. Tests PromQL queries can filter by node labels
+6. Validates example rate() and histogram_quantile() queries work
+
+**Prerequisites:**
+- All Phase 1 prerequisites (see below)
+- Prometheus service enabled with: `docker compose --profile monitoring up -d`
+
+#### Authentication
+
+Tests require authentication tokens for InfluxDB 3 instances. Store tokens in:
+- `~/.env.influxdb3-core-admin-token` (for Core)
+- `~/.env.influxdb3-enterprise-admin-token` (for Enterprise)
+
+Or set environment variables directly:
+- `INFLUXDB3_CORE_TOKEN`
+- `INFLUXDB3_ENTERPRISE_TOKEN`
+
+#### Prerequisites
+
+- Docker and Docker Compose installed
+- Running InfluxDB 3 Core container (`influxdb3-core:8181`)
+- Running InfluxDB 3 Enterprise container (`influxdb3-enterprise:8181`)
+- Valid authentication tokens
+- For Phase 2: Prometheus service (`docker compose --profile monitoring up -d`)
+
 ## Link Validation with Link-Checker
 
 Link validation uses the `link-checker` tool to validate internal and external links in documentation files.

compose.yaml

@@ -306,8 +306,8 @@ services:
     working_dir: /app
   influxdb3-core:
     container_name: influxdb3-core
-    image: influxdb:3-core
-    pull_policy: always
+    image: influxdb:3.5.0-core-arm64
+    pull_policy: never
     # Set variables (except your auth token) for Core in the .env.3core file.
     env_file:
       - .env.3core
@@ -338,8 +338,8 @@ services:
       - influxdb3-core-admin-token
   influxdb3-enterprise:
     container_name: influxdb3-enterprise
-    image: influxdb:3-enterprise
-    pull_policy: always
+    image: influxdb:3.5.0-enterprise-arm64
+    pull_policy: never
     # Set license email and other variables (except your auth token) for Enterprise in the .env.3ent file.
     env_file:
       - .env.3ent
@@ -369,6 +369,74 @@ services:
         target: /var/lib/influxdb3/plugins/custom
     secrets:
       - influxdb3-enterprise-admin-token
+  influxdb3-enterprise-write:
+    container_name: influxdb3-enterprise-write
+    image: influxdb:3.5.0-enterprise-arm64
+    pull_policy: never
+    # Set license email and other variables (except your auth token) for Enterprise in the .env.3ent file.
+    env_file:
+      - .env.3ent
+    ports:
+      - 8183:8181
+    command:
+      - influxdb3
+      - serve
+      - --node-id=writer-0
+      - --mode=ingest
+      - --cluster-id=cluster0
+      - --object-store=file
+      - --data-dir=/var/lib/influxdb3/data
+      - --plugin-dir=/var/lib/influxdb3/plugins
+      - --log-filter=debug
+      - --verbose
+    environment:
+      - INFLUXDB3_AUTH_TOKEN=/run/secrets/influxdb3-enterprise-admin-token
+    volumes:
+      - type: bind 
+        source: test/.influxdb3/enterprise/data
+        target: /var/lib/influxdb3/data
+      - type: bind 
+        source: test/.influxdb3/plugins/influxdata
+        target: /var/lib/influxdb3/plugins
+      - type: bind 
+        source: test/.influxdb3/enterprise/plugins
+        target: /var/lib/influxdb3/plugins/custom
+    secrets:
+      - influxdb3-enterprise-admin-token
+  influxdb3-enterprise-query:
+    container_name: influxdb3-enterprise-query
+    image: influxdb:3.5.0-enterprise-arm64
+    pull_policy: never
+    # Set license email and other variables (except your auth token) for Enterprise in the .env.3ent file.
+    env_file:
+      - .env.3ent
+    ports:
+      - 8184:8181
+    command:
+      - influxdb3
+      - serve
+      - --node-id=querier-0
+      - --mode=query
+      - --cluster-id=cluster0
+      - --object-store=file
+      - --data-dir=/var/lib/influxdb3/data
+      - --plugin-dir=/var/lib/influxdb3/plugins
+      - --log-filter=debug
+      - --verbose
+    environment:
+      - INFLUXDB3_AUTH_TOKEN=/run/secrets/influxdb3-enterprise-admin-token
+    volumes:
+      - type: bind 
+        source: test/.influxdb3/enterprise/data
+        target: /var/lib/influxdb3/data
+      - type: bind 
+        source: test/.influxdb3/plugins/influxdata
+        target: /var/lib/influxdb3/plugins
+      - type: bind 
+        source: test/.influxdb3/enterprise/plugins
+        target: /var/lib/influxdb3/plugins/custom
+    secrets:
+      - influxdb3-enterprise-admin-token
   telegraf-pytest:
     container_name: telegraf-pytest
     image: influxdata/docs-pytest
@@ -499,7 +567,7 @@ services:
   remark-lint:
     container_name: remark-lint
     build:
-      context: . 
+      context: .
       dockerfile: .ci/Dockerfile.remark
     profiles:
       - lint
@@ -510,11 +578,48 @@ services:
       - type: bind
         source: ./CONTRIBUTING.md
         target: /app/CONTRIBUTING.md
+  prometheus:
+    container_name: prometheus
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    environment:
+      - INFLUXDB3_CORE_TOKEN=${INFLUXDB3_CORE_TOKEN}
+      - INFLUXDB3_ENTERPRISE_TOKEN=${INFLUXDB3_ENTERPRISE_TOKEN}
+    volumes:
+      - type: bind
+        source: ./test/influxdb3/prometheus.yml
+        target: /etc/prometheus/prometheus.yml
+        read_only: true
+      - type: volume
+        source: prometheus-data
+        target: /prometheus
+    entrypoint:
+      - /bin/sh
+      - -c
+      - |
+        echo "$$INFLUXDB3_CORE_TOKEN" > /tmp/core-token
+        echo "$$INFLUXDB3_ENTERPRISE_TOKEN" > /tmp/enterprise-token
+        exec /bin/prometheus \
+          --config.file=/etc/prometheus/prometheus.yml \
+          --storage.tsdb.path=/prometheus \
+          --web.console.libraries=/usr/share/prometheus/console_libraries \
+          --web.console.templates=/usr/share/prometheus/consoles \
+          --web.enable-lifecycle
+    depends_on:
+      - influxdb3-core
+      - influxdb3-enterprise
+    profiles:
+      - monitoring
 volumes:
   test-content:
   cloud-tmp:
   cloud-dedicated-tmp:
   cloud-serverless-tmp:
   clustered-tmp:
   telegraf-tmp:
-  v2-tmp:
+  v2-tmp:
+  influxdb3-core-tmp:
+  influxdb2-data:
+  influxdb2-config:
+  prometheus-data:

content/influxdb3/core/admin/monitor-metrics.md

@@ -0,0 +1,24 @@
+---
+title: Monitor metrics
+seotitle: Monitor InfluxDB 3 Core metrics
+description: >
+  Access and understand Prometheus-format metrics exposed by {{< product-name >}}
+  to monitor system performance, resource usage, and operational health.
+menu:
+  influxdb3_core:
+    parent: Administer InfluxDB
+    name: Monitor metrics
+weight: 110
+influxdb3/core/tags: [monitoring, metrics, prometheus, observability, operations]
+related:
+  - /influxdb3/core/reference/internals/runtime-architecture/
+  - /influxdb3/core/admin/performance-tuning/
+  - /influxdb3/core/plugins/library/, InfluxDB 3 Core plugins
+  - /influxdb3/core/write-data/use-telegraf/
+  - /influxdb3/core/reference/telemetry/
+source: /shared/influxdb3-admin/monitor-metrics.md
+---
+
+<!--
+//SOURCE - content/shared/influxdb3-admin/monitor-metrics.md
+-->

content/influxdb3/core/reference/metrics.md

@@ -0,0 +1,22 @@
+---
+title: Metrics
+seotitle: InfluxDB 3 Core metrics reference
+description: >
+  InfluxDB 3 Core exposes Prometheus-format metrics,
+  including descriptions, types, and labels for monitoring and observability.
+menu:
+  influxdb3_core:
+    parent: Reference
+weight: 106
+influxdb3/core/tags: [metrics, prometheus, monitoring, reference, observability]
+related:
+  - /influxdb3/core/admin/monitor-metrics/
+  - /influxdb3/core/reference/telemetry/
+  - /influxdb3/core/reference/internals/runtime-architecture/
+source: /shared/influxdb3-reference/metrics.md
+---
+
+<!--
+The content of this file is located at
+//SOURCE - content/shared/influxdb3-reference/metrics.md
+-->

content/influxdb3/enterprise/admin/last-value-cache/_index.md

@@ -9,7 +9,7 @@ description: >
 menu:
   influxdb3_enterprise:
     parent: Administer InfluxDB
-weight: 105
+weight: 106
 influxdb3/enterprise/tags: [cache]
 related:
   - /influxdb3/enterprise/reference/sql/functions/cache/#last_cache, last_cache SQL function

content/influxdb3/enterprise/admin/monitor-metrics.md

@@ -0,0 +1,26 @@
+---
+title: Monitor metrics
+seotitle: Monitor {{< product-name >}} metrics
+description: >
+  Access and understand Prometheus-format metrics exposed by {{< product-name >}}
+  to monitor distributed cluster performance, resource usage, and operational health.
+menu:
+  influxdb3_enterprise:
+    parent: Administer InfluxDB
+    name: Monitor metrics
+weight: 110
+influxdb3/enterprise/tags: [monitoring, metrics, prometheus, observability, operations, clustering]
+related:
+  - /influxdb3/enterprise/admin/clustering/
+  - /influxdb3/enterprise/reference/internals/runtime-architecture/
+  - /influxdb3/enterprise/admin/performance-tuning/
+  - /influxdb3/enterprise/plugins/library/, InfluxDB 3 Enterprise plugins
+  - /influxdb3/enterprise/write-data/use-telegraf/
+  - /influxdb3/enterprise/reference/telemetry/
+source: /shared/influxdb3-admin/monitor-metrics.md
+---
+
+<!--
+The content of this file is located at
+//SOURCE - content/shared/influxdb3-admin/monitor-metrics.md
+-->