From 12ce05db98ef197a6d9b30502dfc40589372f472 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 14:31:44 +0200 Subject: [PATCH 01/14] corrected docker compose syntax. Folder is not specified this way anymore, and we don't actually need to specify it. --- modules/ROOT/pages/installation/docker/compose.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/installation/docker/compose.adoc b/modules/ROOT/pages/installation/docker/compose.adoc index c94106e1..3457873d 100644 --- a/modules/ROOT/pages/installation/docker/compose.adoc +++ b/modules/ROOT/pages/installation/docker/compose.adoc @@ -63,7 +63,7 @@ NOTE: Self signed certificates can be generated within the container using a ser + [source, shell, role=noheader] ---- -docker compose -f up +docker compose -f up ---- ==== From 2d70f26af9c18d4d5d81e30d41db29e4765f53c2 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 14:32:21 +0200 Subject: [PATCH 02/14] made URL point to Ops Manager artifacts --- modules/ROOT/pages/installation/kubernetes/helm-charts.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/installation/kubernetes/helm-charts.adoc b/modules/ROOT/pages/installation/kubernetes/helm-charts.adoc index d31a6e0a..aeaa19db 100644 --- a/modules/ROOT/pages/installation/kubernetes/helm-charts.adoc +++ b/modules/ROOT/pages/installation/kubernetes/helm-charts.adoc @@ -9,7 +9,7 @@ - Access to a Kubernetes enviroment (cloud, on-prem, or local with a `LoadBalancer` resource implementation). == NOM server Helm Chart -* Download the NOM server Helm Chart from link:https://neo4j.com/download-center/[Neo4j Download Center]. +* Download the NOM server Helm Chart from link:https://neo4j.com/download-center/#ops-manager[Neo4j Download Center]. * Following is the reference `values.yaml` for NOM server Helm Chart: ---- From d5b6fdfa3db37fabc5d61fac2a2c7d6f275c7fb2 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 14:32:32 +0200 Subject: [PATCH 03/14] made URL point to Ops Manager artifacts --- modules/ROOT/pages/first-look/linux-first-look.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/first-look/linux-first-look.adoc b/modules/ROOT/pages/first-look/linux-first-look.adoc index ebaeddb7..1dd61b6d 100644 --- a/modules/ROOT/pages/first-look/linux-first-look.adoc +++ b/modules/ROOT/pages/first-look/linux-first-look.adoc @@ -9,7 +9,7 @@ NOTE: The query log feature is only available for DBMSs with instances that are == Create persistence DBMS === Download and extract -Download latest Neo4j Linux/Mac executable from https://neo4j.com/download-center/#ops-manager[here] and extract the package. +Download latest Neo4j Linux/Mac executable from https://neo4j.com/download-center/#current-releases[here] and extract the package. Extracted directory location is referred to as NEO4J_HOME form now on. === Change password From 536511efbea15aa09f4ed66c44739d194ec409ce Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 14:39:16 +0200 Subject: [PATCH 04/14] improved description of GRPC_SERVER_SECURITY_CLIENT_AUTH --- modules/ROOT/pages/installation/server.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/pages/installation/server.adoc b/modules/ROOT/pages/installation/server.adoc index 1f7f577b..e80d7211 100644 --- a/modules/ROOT/pages/installation/server.adoc +++ b/modules/ROOT/pages/installation/server.adoc @@ -311,7 +311,7 @@ If the NOM Server is required to support self-registered agents, then additional | `grpc.server.security.clientAuth` | `GRPC_SERVER_SECURITY_CLIENT_AUTH` -| Option to indicate whether to verify client certificates. Required for agent self-registration. (optional) +| Option to indicate whether to verify client certificates. Required for agent self-registration. Possible values: `OPTIONAL` to simultaneously allow manual registration or `REQUIRE` to only allow mutually authenticated agents. | `OPTIONAL` | `logging.file.name` From b95b702372e87b57feed3e4ef537147d91c87e74 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 14:51:15 +0200 Subject: [PATCH 05/14] updated to use openssl for certificate creation, mTLS and agent self-registration --- .../pages/first-look/docker-first-look.adoc | 163 ++++++++++-------- 1 file changed, 95 insertions(+), 68 deletions(-) diff --git a/modules/ROOT/pages/first-look/docker-first-look.adoc b/modules/ROOT/pages/first-look/docker-first-look.adoc index ea9a7971..d11cb0b4 100644 --- a/modules/ROOT/pages/first-look/docker-first-look.adoc +++ b/modules/ROOT/pages/first-look/docker-first-look.adoc @@ -6,8 +6,6 @@ Docker compose can be used to provision a set of related containers as services Docker does not recommend using compose for production. -The manual installation, configuring and running of agents on the Neo4j containers is lost if the container is stopped. - == Prerequisites * Unix-based operating system * Docker @@ -15,36 +13,48 @@ The manual installation, configuring and running of agents on the Neo4j containe == Prepare hosts file -You need to set up the following host names in your `/etc/hosts` or `C:/windows/system32/drivers/etc/hosts file`; +You need to set up the following host names in your `/etc/hosts` file: [source,, role=noheader] ---- 127.0.0.1 storage server db-single ---- -== Create persistence DBMS and NOM server - -=== Generate self-signed certificates +== Generate self-signed certificates . Create a directory *`~/.nom/ssc`* on your local machine before running Docker compose. The compose files below specify that this directory will be mounted into the docker container to make the generated certificates available to the NOM server and the agent. +. Change to *`~/.nom/ssc`* and generate self-signed certificates using `server` as common name (CN) and DNS name, and `127.0.0.1` as IP address (details see *xref:installation/self-signed-certificate.adoc[here]*): + +[source, shell] +---- +openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ + -subj "/CN=server" \ + -addext "subjectAltName = DNS:server, IP:127.0.0.1" \ + -addext "keyUsage = critical, digitalSignature, keyEncipherment" \ + -addext "extendedKeyUsage = serverAuth" \ + -addext "authorityKeyIdentifier = keyid:always,issuer:always" \ + -keyout "server.key" \ + -out "server.cer" + +openssl pkcs12 -export \ + -inkey "server.key" \ + -in "server.cer" \ + -out "server.pfx" \ + -password "pass:changeit" +---- +This results in the creation of `server.cer`, `server.key` and `server.pfx`. +. Use the password applied for the certificate store (`changeit`) instead of `` in the next sections. + [NOTE] ==== -For *`sudo`* docker compose command, make sure the correct path to *`.nom/ssc`* is set in the bind section below. +To keep this example simple, the generated certificates are used for both NOM server and agents. ==== -. Change to *`~/.nom/ssc`* and generate self-signed certificates as described *xref:installation/self-signed-certificate.adoc[here]*, using `server` as common name (CN) and DNS name, and `127.0.0.1` as IP address. -This results in the creation of `server.cer` and `server.pfx`. -. Use the password applied for the certificate store instead of `` in the next sections. - -=== Copy docker compose file -Following is a docker compose file that can be used to start up a NOM server and Enerprise Neo4j Persistence system without any customization. -Copy this to your local machine as `docker-compose.yaml`. +== Create the docker compose file -Edit the yaml as follows: - -* replace all three occurrences of with a secure password. -* replace all three occurrences of with a secure password. +Following is a docker compose file that can be used to start up a NOM server including its persistence and a single Neo4j Enterprise instance that will be managed by NOM. +Create the file `docker-compose.yaml` in an empty directory on your local machine. +This directory is referenced as `INSTALLATION_ROOT` later in this document. .docker-compose.yaml [source, yaml] @@ -55,7 +65,7 @@ networks: services: storage: hostname: storage - image: neo4j:4.4.8-enterprise + image: neo4j:enterprise networks: - lan ports: @@ -93,6 +103,8 @@ services: GRPC_SERVER_SECURITY_KEY_STORE_PASSWORD: CORS_ALLOWEDHEADERS: "*" CORS_ALLOWEDORIGINS: "http://localhost:[*],https://localhost:[*], http://server:[*]" + GRPC_SERVER_SECURITY_TRUST_CERT_COLLECTION: file:/certificates/server.cer + GRPC_SERVER_SECURITY_CLIENT_AUTH: OPTIONAL volumes: - type: bind source: ~/.nom/ssc @@ -101,55 +113,20 @@ services: - "sh" - "-c" - "java -jar app.jar" ----- - -Documentation for NOM server docker image is *xref:installation/docker/container.adoc[here]*. - -=== Run docker compose -Run the follwing command: - -[source, shell] ----- -docker compose -f docker-compose.yaml up ----- - -=== Open NOM UI -Wait for the server container to start and then go to https://server:8080. -Login as admin:passw0rd and accept license terms. - -== Set up instance and agent - -=== Register agent -In NOM UI - navigate to Agent settings (clicking on the sad robot takes you to the correct page) and add a new agent. -Copy environment variables that are provided. - -Full documentation on registering an agent is *xref:addition/index.adoc#register[here]*. - -=== Create docker compose file for Neo4j instance (pre-configured for agent) - -Copy compose file below into `docker-compose.instance.yaml` and edit as follows: - -* Replace `` and `` with the values shown during the reigister agent step in the NOM UI. -* Replace all three occurrences of with a secure password. - -.docker-compose.instance.yaml -[source, yaml] ----- -services: db-single: hostname: db-single - image: neo4j:5.5.0-enterprise + image: neo4j:enterprise networks: - lan ports: - "10000:10000" - "10001:10001" environment: - CONFIG_SERVER_ADDRESS: "server:9090" - CONFIG_TOKEN_URL: "https://server:8080/api/login/agent" - CONFIG_TOKEN_CLIENT_ID: "" - CONFIG_TOKEN_CLIENT_SECRET: "" + CONFIG_SERVER_GRPC_ADDRESS: "server:9090" + CONFIG_SERVER_HTTP_ADDRESS: "https://server:8080" CONFIG_TLS_TRUSTED_CERTS: "/certificates/server.cer" + CONFIG_TLS_CLIENT_CERT: "/certificates/server.cer" + CONFIG_TLS_CLIENT_KEY: "/certificates/server.key" CONFIG_LOG_LEVEL: "debug" CONFIG_INSTANCE_1_NAME: "single-instance" CONFIG_INSTANCE_1_BOLT_URI: "bolt://db-single:10001" @@ -170,8 +147,11 @@ services: NEO4J_server_metrics_filter: "*" volumes: - type: bind - source: ~/.nom/ssc - target: /certificates + source: ~/.nom/ssc + target: /certificates + - type: bind + source: agent + target: /agent healthcheck: test: [ "CMD-SHELL", "echo RETURN 1 | cypher-shell -a bolt://db-single:10001 -u neo4j -p || exit 1" ] interval: 10s @@ -180,23 +160,70 @@ services: start_period: 5s ---- -=== Run docker compose -Run the follwing command: +Documentation for NOM server docker image is *xref:installation/docker/container.adoc[here]*. + +Edit `docker-compose.yaml` as follows: + +* Replace all occurrences of `SSC_PASSWORD` with the certificate store password applied above. +* Replace all occurrences of `PERSISTENCE_PASSWORD` with a secure password. +* Replace all occurrences of `NEO4J_INSTANCE_PASSWORD` with a secure password. + +== Start the docker compose environment +Run the following command in your `INSTALLATION_ROOT`: [source, shell] ---- -docker compose -f docker-compose.instance.yaml up +docker compose -f docker-compose.yaml up ---- -=== Run agent +Watch the output and make sure that docker containers `storage`, `server` and `db-single` are started successfully. + +== Download and start the NOM agent +* Download NOM agent binaries TAR from https://neo4j.com/download-center/#ops-manager[here] and execute the following commands in your `INSTALLATION_ROOT`: ++ [source, shell] ---- -docker compose -f docker-compose.instance.yaml exec sh -c tar -xvf products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz && neo4j-ops-manager-agent-*/bin/agent console +mkdir agent +tar -xvf -C agent --strip-components=1 ---- +* In `INSTALLATION_ROOT`, start the agent in self-registration mode: ++ +[source, shell] +---- +docker compose -f docker-compose.yaml exec db-single sh -c "/agent/bin/agent console -s" +---- + +Full documentation on registering an agent is *xref:addition/agent-installation/index.adoc[here]*. == Explore NOM UI -Go to NOM UI and wait for DBMS to appear - this may take a few minutes. -You should be able to see that the agent has connected in the agents listing. + +* Wait for the server container to start and then go to https://server:8080. +* Login as `admin` / `passw0rd` and accept license terms. +* Click the top right settings icon that redirects you to the global settings. +* Make sure that the agent is online. Rename the agent if required. +* Return to the main page and wait for DBMS to appear - this may take a few minutes. Once the DBMS is shown in the home page, double-click on the name (initially a generated string) to edit it. Double-click on the DBMS to see the metrics, status, security panel, logs and upgrade pages for the DBMS. + +== Controlling docker containers + +=== Stopping + +* To stop the complete NOM environment, press `Ctrl-C` on the docker compose console and the agent console. +* To stop a single docker container, issue `docker container stop `. +To list containers use `docker ps`. + +[NOTE] +==== +Since docker keeps persisted data in container volumes, restarted containers will keep the previous state. +==== + +=== Resetting + +To start over with an empty Neo4j persistence and empty managed instance, use the following docker command (use `docker ps -a` to find out actual container names): + +[source, shell] +---- +docker container rm -v +---- From 2839dce5ff73b34d5e5bf22721b95fc789b9520d Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 15:03:52 +0200 Subject: [PATCH 06/14] corrected values to be replaced --- modules/ROOT/pages/first-look/docker-first-look.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/first-look/docker-first-look.adoc b/modules/ROOT/pages/first-look/docker-first-look.adoc index d11cb0b4..fcb1912f 100644 --- a/modules/ROOT/pages/first-look/docker-first-look.adoc +++ b/modules/ROOT/pages/first-look/docker-first-look.adoc @@ -164,9 +164,9 @@ Documentation for NOM server docker image is *xref:installation/docker/container Edit `docker-compose.yaml` as follows: -* Replace all occurrences of `SSC_PASSWORD` with the certificate store password applied above. -* Replace all occurrences of `PERSISTENCE_PASSWORD` with a secure password. -* Replace all occurrences of `NEO4J_INSTANCE_PASSWORD` with a secure password. +* Replace all occurrences of `` with the certificate store password applied above. +* Replace all occurrences of `` with a secure password. +* Replace all occurrences of `` with a secure password. == Start the docker compose environment Run the following command in your `INSTALLATION_ROOT`: From b272a60025704f49f56c6a2b8d71aed828f1129f Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Mon, 25 Sep 2023 15:23:04 +0200 Subject: [PATCH 07/14] typos --- modules/ROOT/pages/addition/kubernetes/sidecar.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/addition/kubernetes/sidecar.adoc b/modules/ROOT/pages/addition/kubernetes/sidecar.adoc index bbc092c8..a3b384fa 100644 --- a/modules/ROOT/pages/addition/kubernetes/sidecar.adoc +++ b/modules/ROOT/pages/addition/kubernetes/sidecar.adoc @@ -1,11 +1,11 @@ = NOM agent as a Sidecar -A NOM agent can be deployed as a Sidecar container within the same pod as Neo4j instance in a Kubernetes cluster. Agent sidecar does not need to use a official agent image. +A NOM agent can be deployed as a Sidecar container within the same pod as Neo4j instance in a Kubernetes cluster. Agent sidecar does not need to use an official agent image. An agent can simply be run in a lightweight linux based container such as `alpine`. The agent binary can be either mounted into container or can be extracted from Neo4j image using an init container. [WARNING] ==== -An agent Sidecar container can not access all resources that Neo4j pod uses and the host metrics of the cluster node. For example, for disk usage metrics, agent needs access to file system or peristent volumes that Neo4j pod uses. These need to be mounted to agent Daemon pod. Agent sidecar can only report host metrics of it's own contained environment which is not a complete view of reosurce usage of cluster node. +An agent Sidecar container can not access all resources that Neo4j pod uses and the host metrics of the cluster node. For example, for disk usage metrics, agent needs access to file system or peristent volumes that Neo4j pod uses. These need to be mounted to agent Daemon pod. Agent sidecar can only report host metrics of its own contained environment which is not a complete view of resource usage of cluster node. ==== Following is a sample `values.yaml` file for Neo4j Helm charts with NOM agent sidecar configured: @@ -85,4 +85,4 @@ podSpec: volumeMounts: - name: agent-bin mountPath: /agent/bin ----- \ No newline at end of file +---- From e5120b2603f319fbc58f460bb1fa43211096c5c7 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 09:11:22 +0200 Subject: [PATCH 08/14] typos --- .../agent-installation/agent-self-signed-certificates.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/addition/agent-installation/agent-self-signed-certificates.adoc b/modules/ROOT/pages/addition/agent-installation/agent-self-signed-certificates.adoc index a5c3f2bc..e543ab7e 100644 --- a/modules/ROOT/pages/addition/agent-installation/agent-self-signed-certificates.adoc +++ b/modules/ROOT/pages/addition/agent-installation/agent-self-signed-certificates.adoc @@ -15,11 +15,11 @@ The NOM agent contains a utility to generate a self-signed certificate suitable ---- $> agent ssc --help -utility to generate self signed certificate for TEST purposes only +utility to generate self-signed certificate for TEST purposes only sample usage: -to generate a self signed certificate for 'localhost', which could either be +to generate a self-signed certificate for 'localhost', which could either be accessed through DNS names of 'localhost.localdomain' or 'my.custom.domain', or with IP addresses of '127.0.0.1' or '192.168.100.5'; From dbf73f0d5259d9cf751e2dcc519d1cd5e5e15df6 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 09:37:43 +0200 Subject: [PATCH 09/14] made more compact without loosing information --- .../pages/installation/docker/container.adoc | 49 +++++-------------- 1 file changed, 12 insertions(+), 37 deletions(-) diff --git a/modules/ROOT/pages/installation/docker/container.adoc b/modules/ROOT/pages/installation/docker/container.adoc index 06e3271b..bed036ce 100644 --- a/modules/ROOT/pages/installation/docker/container.adoc +++ b/modules/ROOT/pages/installation/docker/container.adoc @@ -13,46 +13,18 @@ docker pull neo4j/neo4j-ops-manager-server:latest NOM server in a Docker container can be run in several contexts such as a Docker compose environment or a in a Kubernetes cluster etc. In this chapter, we only focus on running a standalone NOM server in a Docker container (Henceforth referred as NOM server container). -== Configuration -NOM server container run configuration involves setting required environment variables or passing them as commandline arguments. - -Refer to *xref:../installation/server.adoc#config_ref[Server installation]* for the list of configuration options to be provided to NOM server. - -== Security -NOM server security involves configuring keys and certificates for the TLS connections that the server makes with the UI and the agents. -For NOM server container, keys, and certificates should be mounted into the container as shown below: - -. As command line options -+ -[source, shell, role=noheader] ----- -docker run -v : \ - neo4j/neo4j-ops-manager-server \ - --server.ssl.key-store= \ - --grpc.server.security.key-store= ----- - -. As environment variables -+ -[source, shell, role=noheader] ----- -docker run -v : \ - neo4j/neo4j-ops-manager-server \ - -e "SERVER_SSL_KEY_STORE=" \ - -e "GRPC_SERVER_SECURITY_KEY_STORE=" ----- - -NOTE: NOM server also includes a command to generate self-signed certificates which can be used for testing purpose but not recommended for production. -This command could be run within the container and generate the certificates for the server. -Refer to *xref:../installation/self-signed-certificate.adoc[Installation > Self-Signed-Certificate]* - == Running a NOM server in a Docker container -To successfully start the NOM server container, mandatory runtime parameters should be set. There are two ways these parameters can be provided to the container: -IMPORTANT: Make sure to replace the arguments with values adjusted to your environment (**). -Refer to *xref:../installation/server.adoc#_unix[Installation > Server Installation]* +[NOTE] +==== +Securing NOM server involves configuring keys and certificates for the TLS connections that the server makes with the UI and the agents. +For testing purposes, self-signed certificates can be used. +Refer to *xref:../installation/self-signed-certificate.adoc[Installation > Self-Signed-Certificate]* on how to generate self-signed certificates. +==== -IMPORTANT: Make sure ports are set as per the below commands. +To successfully start the NOM server container, mandatory runtime parameters should be set. +Refer to *xref:../installation/server.adoc#config_ref[Server installation]* for the complete parameter list. +There are two ways these parameters can be provided to the container: . As command line options + @@ -94,5 +66,8 @@ docker run \ neo4j/neo4j-ops-manager-server ---- +IMPORTANT: Make sure to replace the arguments with values adjusted to your environment (**). +Refer to *xref:../installation/server.adoc#_unix[Installation > Server Installation]* + == Maintenance For a standalone NOM server container resource constraints can be set as documented by link:https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources[Docker runtime constraints on resources]. From 7d3ab9f3df83bd864aaf9ade57ee913a6e34136b Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 09:48:01 +0200 Subject: [PATCH 10/14] refer to docker first look guide for a concrete example to avoid duplication --- .../pages/installation/docker/compose.adoc | 78 ++----------------- 1 file changed, 5 insertions(+), 73 deletions(-) diff --git a/modules/ROOT/pages/installation/docker/compose.adoc b/modules/ROOT/pages/installation/docker/compose.adoc index 3457873d..8699d6d8 100644 --- a/modules/ROOT/pages/installation/docker/compose.adoc +++ b/modules/ROOT/pages/installation/docker/compose.adoc @@ -5,7 +5,7 @@ Docker compose can be used to provision a set of related containers as services But Docker doesn't recommend using compose for production. -Following are a set of steps to create a two container compose setup to experiment with NOM server container and an Enterprise Neo4j persistence container as storage: +Following is a set of steps to create a two container compose setup to experiment with NOM server container and an Enterprise Neo4j persistence container as storage: * Define a storage service that uses an Enterprise Neo4j image. Follow the below template and replace appropriate user defined values. + @@ -45,20 +45,18 @@ storage: SPRING_NEO4J_AUTHENTICATION_USERNAME: SPRING_NEO4J_AUTHENTICATION_PASSWORD: SERVER_SSL_KEY_STORE_TYPE: PKCS12 - SERVER_SSL_KEY_STORE: + SERVER_SSL_KEY_STORE: SERVER_SSL_KEY_STORE_PASSWORD: GRPC_SERVER_SECURITY_KEY_STORE_TYPE: PKCS12 - GRPC_SERVER_SECURITY_KEY_STORE: + GRPC_SERVER_SECURITY_KEY_STORE: GRPC_SERVER_SECURITY_KEY_STORE_PASSWORD: GRPC_SERVER_SECURITY_CLIENT_AUTH: "OPTIONAL" CORS_ALLOWEDHEADERS: "*" CORS_ALLOWEDORIGINS: "http://localhost:[*],https://localhost:[*]" volumes: - - : + - : ---- + -NOTE: Self signed certificates can be generated within the container using a server sub command as mentioned here Refer to *xref:../installation/self-signed-certificate.adoc[Installation > Self-Signed-Certificate]*. In this case mounting the certs into the container is not required. - * Once the services are defined and saved to a compose file, run the following command: + [source, shell, role=noheader] @@ -66,70 +64,4 @@ NOTE: Self signed certificates can be generated within the container using a ser docker compose -f up ---- -==== -Following is an example Docker compose file that can be used to start up a NOM server and Enterprise Neo4j Persistence system. You can use it to quickly setup a NOM test environment and add DBMSs for monitoring and customize it with additional configuration: - -IMPORTANT: Create a directory *`~/.nom/ssc`* before running Docker compose. - -NOTE: Certificates are generated as by-products of running below compose file at *`~/.nom/ssc`* directory which can be used to configure the NOM agent as given xref:addition/index.adoc#configure[here]. - -NOTE: For *`sudo`* docker compose command, make sure the correct path to *`.nom/ssc`* is set in the bind section below. - -.docker-compose.yaml -[source, yaml] ----- -networks: - lan: - -services: - storage: - hostname: storage - image: neo4j:enterprise - networks: - - lan - ports: - - "9000:9000" - - "9001:9001" - environment: - NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes" - NEO4J_AUTH: neo4j/passw0rd - NEO4J_dbms_default__advertised__address: storage - NEO4J_dbms_connector_http_listen__address: storage:9000 - NEO4J_dbms_connector_bolt_listen__address: storage:9001 - healthcheck: - test: [ "CMD-SHELL", "echo RETURN 1 | cypher-shell -a bolt://storage:9001 -u neo4j -p passw0rd || exit 1" ] - - server: - hostname: server - image: neo4j/neo4j-ops-manager-server:latest - depends_on: - storage: - condition: service_healthy - networks: - - lan - ports: - - "8080:8080" - - "9090:9090" - environment: - SPRING_NEO4J_URI: bolt://storage:9001 - SPRING_NEO4J_AUTHENTICATION_USERNAME: neo4j - SPRING_NEO4J_AUTHENTICATION_PASSWORD: passw0rd - SERVER_SSL_KEY_STORE_TYPE: PKCS12 - SERVER_SSL_KEY_STORE: file:/certificates/localhost.pfx - SERVER_SSL_KEY_STORE_PASSWORD: changeit - GRPC_SERVER_SECURITY_KEY_STORE_TYPE: PKCS12 - GRPC_SERVER_SECURITY_KEY_STORE: file:/certificates/localhost.pfx - GRPC_SERVER_SECURITY_KEY_STORE_PASSWORD: changeit - GRPC_SERVER_SECURITY_CLIENT_AUTH: "OPTIONAL" - CORS_ALLOWEDHEADERS: "*" - CORS_ALLOWEDORIGINS: "http://localhost:[*],https://localhost:[*]" - volumes: - - type: bind - source: ~/.nom/ssc - target: /certificates - entrypoint: - - "sh" - - "-c" - - "java -jar app.jar ssc -n localhost -o /certificates -p changeit -d localhost.localdomain -i 127.0.0.1 && java -jar app.jar" ----- -==== +Please refer to xref:first-look/docker-first-look.adoc[Docker first look guide] for a complete example of a docker compose environment with NOM persistence, NOM server and a managed Neo4j instance. From 5efe19b1b5bf76361087aac64cab2dd1bc198d9d Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 09:50:22 +0200 Subject: [PATCH 11/14] moved reference to the example to be a note in front of the steps --- modules/ROOT/pages/installation/docker/compose.adoc | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/installation/docker/compose.adoc b/modules/ROOT/pages/installation/docker/compose.adoc index 8699d6d8..a96ed877 100644 --- a/modules/ROOT/pages/installation/docker/compose.adoc +++ b/modules/ROOT/pages/installation/docker/compose.adoc @@ -5,7 +5,12 @@ Docker compose can be used to provision a set of related containers as services But Docker doesn't recommend using compose for production. -Following is a set of steps to create a two container compose setup to experiment with NOM server container and an Enterprise Neo4j persistence container as storage: +Following is a set of steps to create a two container compose setup to experiment with NOM server container and an Enterprise Neo4j persistence container as storage. + +[NOTE] +==== +Please refer to xref:first-look/docker-first-look.adoc[Docker first look guide] for a complete example of a docker compose environment with NOM persistence, NOM server and a managed Neo4j instance. +==== * Define a storage service that uses an Enterprise Neo4j image. Follow the below template and replace appropriate user defined values. + @@ -63,5 +68,3 @@ storage: ---- docker compose -f up ---- - -Please refer to xref:first-look/docker-first-look.adoc[Docker first look guide] for a complete example of a docker compose environment with NOM persistence, NOM server and a managed Neo4j instance. From d599480acb839227ea128bfd4526a41cc77fbdb2 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 10:08:43 +0200 Subject: [PATCH 12/14] updated to use 5.x Neo4j setting names --- .../ROOT/pages/addition/docker/container.adoc | 27 ++++++++++--------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/modules/ROOT/pages/addition/docker/container.adoc b/modules/ROOT/pages/addition/docker/container.adoc index ef080253..09ebe7c9 100644 --- a/modules/ROOT/pages/addition/docker/container.adoc +++ b/modules/ROOT/pages/addition/docker/container.adoc @@ -1,15 +1,16 @@ = Monitor a Neo4j instance in a standalone Docker container -Running additional processes which are not tied to the lifecycle of container process is not a best practice for containerized applications. Nevertheless, it is possible transparently monitor a containerized Neo4j instance in following ways: +Running additional processes which are not tied to the lifecycle of container process is not a best practice for containerized applications. +Nevertheless, it is possible to transparently monitor a containerized Neo4j instance in following ways: [NOTE] ==== -All methods below use a bundled NOM agent inside the Neo4j image instead of downloaded agent packages. Replacing `products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz` with +Both methods below use a bundled NOM agent inside the Neo4j image instead of downloaded agent packages. Replacing `products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz` with path to NOM agent package downloaded should give the same results. ==== -. Monitoring a Neo4j instance Running inside a Docker container; - +. Monitoring a Neo4j instance Running inside a Docker container: ++ [source, shell, role=noheader] ---- docker exec -dit /bin/sh 'tar -xzf products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz --strip-components 1 \ @@ -29,14 +30,14 @@ CONFIG_INSTANCE_1_QUERY_LOG_MIN_DURATION="100" \ NEO4J_ACCEPT_LICENSE_AGREEMENT="yes" \ NEO4J_AUTH=neo4j/passw0rd \ NEO4J_EDITION="enterprise" \ -NEO4J_metrics_prometheus_enabled="true" \ -NEO4J_metrics_prometheus_endpoint="localhost:2004" \ -NEO4J_metrics_filter="*" \ +NEO4J_server_metrics_prometheus_enabled="true" \ +NEO4J_server_metrics_prometheus_endpoint="localhost:2004" \ +NEO4J_server_metrics_filter="*" \ agent console -s' ---- -. Start a Neo4j instance Docker container with NOM agent using a custom entry point; - +. Start a Neo4j instance Docker container with NOM agent using a custom entry point: ++ [[entrypoint]] .entrypoint.sh [source, shell] @@ -61,12 +62,12 @@ CONFIG_INSTANCE_1_QUERY_LOG_MIN_DURATION="100" \ NEO4J_ACCEPT_LICENSE_AGREEMENT="yes" \ NEO4J_AUTH=neo4j/passw0rd \ NEO4J_EDITION="enterprise" \ -NEO4J_metrics_prometheus_enabled="true" \ -NEO4J_metrics_prometheus_endpoint="localhost:2004" \ -NEO4J_metrics_filter="*" \ +NEO4J_server_metrics_prometheus_enabled="true" \ +NEO4J_server_metrics_prometheus_endpoint="localhost:2004" \ +NEO4J_server_metrics_filter="*" \ agent console -s ---- - ++ [source, shell, role=noheader] ---- docker run --entrypoint /custom/entrypoint.sh -e NEO4J_ACCEPT_LICENSE_AGREEMENT=yes -v :/custom $NEO4J_IMAGE From cb4e92368152dd83e43adf67a27c1a3bf3f47281 Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 13:11:21 +0200 Subject: [PATCH 13/14] Apply suggestions from code review Co-authored-by: Jessica Wright <49636617+AlexicaWright@users.noreply.github.com> --- .../ROOT/pages/addition/docker/container.adoc | 6 ++--- .../pages/addition/kubernetes/sidecar.adoc | 11 +++++--- .../pages/first-look/docker-first-look.adoc | 25 ++++++++++--------- .../pages/installation/docker/compose.adoc | 5 ++-- .../pages/installation/docker/container.adoc | 2 +- modules/ROOT/pages/installation/server.adoc | 4 ++- 6 files changed, 31 insertions(+), 22 deletions(-) diff --git a/modules/ROOT/pages/addition/docker/container.adoc b/modules/ROOT/pages/addition/docker/container.adoc index 09ebe7c9..5f310dcd 100644 --- a/modules/ROOT/pages/addition/docker/container.adoc +++ b/modules/ROOT/pages/addition/docker/container.adoc @@ -1,12 +1,12 @@ = Monitor a Neo4j instance in a standalone Docker container -Running additional processes which are not tied to the lifecycle of container process is not a best practice for containerized applications. +Running additional processes that are not tied to the lifecycle of the container process is not a best practice for containerized applications. Nevertheless, it is possible to transparently monitor a containerized Neo4j instance in following ways: [NOTE] ==== -Both methods below use a bundled NOM agent inside the Neo4j image instead of downloaded agent packages. Replacing `products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz` with -path to NOM agent package downloaded should give the same results. +Both methods below use a bundled NOM agent inside the Neo4j image instead of downloaded agent packages. +Replacing `products/neo4j-ops-manager-agent-*-linux-amd64.tar.gz` with path to NOM agent package downloaded should give the same results. ==== . Monitoring a Neo4j instance Running inside a Docker container: diff --git a/modules/ROOT/pages/addition/kubernetes/sidecar.adoc b/modules/ROOT/pages/addition/kubernetes/sidecar.adoc index a3b384fa..d8e9909b 100644 --- a/modules/ROOT/pages/addition/kubernetes/sidecar.adoc +++ b/modules/ROOT/pages/addition/kubernetes/sidecar.adoc @@ -1,11 +1,16 @@ = NOM agent as a Sidecar -A NOM agent can be deployed as a Sidecar container within the same pod as Neo4j instance in a Kubernetes cluster. Agent sidecar does not need to use an official agent image. -An agent can simply be run in a lightweight linux based container such as `alpine`. The agent binary can be either mounted into container or can be extracted from Neo4j image using an init container. +A NOM agent can be deployed as a Sidecar container within the same pod as Neo4j instance in a Kubernetes cluster. +The agent Sidecar does not need to use an official agent image. +An agent can simply be run in a lightweight Linux-based container such as `alpine`. +The agent binary can be either mounted into a container or can be extracted from the Neo4j image using an init container. [WARNING] ==== -An agent Sidecar container can not access all resources that Neo4j pod uses and the host metrics of the cluster node. For example, for disk usage metrics, agent needs access to file system or peristent volumes that Neo4j pod uses. These need to be mounted to agent Daemon pod. Agent sidecar can only report host metrics of its own contained environment which is not a complete view of resource usage of cluster node. +An agent Sidecar container cannot access all resources that the Neo4j pod uses and the host-metrics of the cluster node. +For example, for disk usage metrics, the agent needs access to the file system or persistent volumes that the Neo4j pod uses. +These need to be mounted to the agent Daemon pod. +The agent Sidecar can only report host metrics of its own contained environment which is not a complete view of the resource usage of the cluster node. ==== Following is a sample `values.yaml` file for Neo4j Helm charts with NOM agent sidecar configured: diff --git a/modules/ROOT/pages/first-look/docker-first-look.adoc b/modules/ROOT/pages/first-look/docker-first-look.adoc index fcb1912f..150b5c53 100644 --- a/modules/ROOT/pages/first-look/docker-first-look.adoc +++ b/modules/ROOT/pages/first-look/docker-first-look.adoc @@ -22,7 +22,7 @@ You need to set up the following host names in your `/etc/hosts` file: == Generate self-signed certificates . Create a directory *`~/.nom/ssc`* on your local machine before running Docker compose. -The compose files below specify that this directory will be mounted into the docker container to make the generated certificates available to the NOM server and the agent. +The compose files below specify that this directory will be mounted into the Docker container to make the generated certificates available to the NOM server and the agent. . Change to *`~/.nom/ssc`* and generate self-signed certificates using `server` as common name (CN) and DNS name, and `127.0.0.1` as IP address (details see *xref:installation/self-signed-certificate.adoc[here]*): + [source, shell] @@ -50,9 +50,9 @@ This results in the creation of `server.cer`, `server.key` and `server.pfx`. To keep this example simple, the generated certificates are used for both NOM server and agents. ==== -== Create the docker compose file +== Create the Docker compose file -Following is a docker compose file that can be used to start up a NOM server including its persistence and a single Neo4j Enterprise instance that will be managed by NOM. +Following is a Docker compose file that can be used to start up a NOM server including its persistence and a single Neo4j Enterprise instance that will be managed by NOM. Create the file `docker-compose.yaml` in an empty directory on your local machine. This directory is referenced as `INSTALLATION_ROOT` later in this document. @@ -160,7 +160,7 @@ services: start_period: 5s ---- -Documentation for NOM server docker image is *xref:installation/docker/container.adoc[here]*. +Documentation for NOM server Docker image is *xref:installation/docker/container.adoc[here]*. Edit `docker-compose.yaml` as follows: @@ -168,7 +168,7 @@ Edit `docker-compose.yaml` as follows: * Replace all occurrences of `` with a secure password. * Replace all occurrences of `` with a secure password. -== Start the docker compose environment +== Start the Docker compose environment Run the following command in your `INSTALLATION_ROOT`: [source, shell] @@ -176,7 +176,7 @@ Run the following command in your `INSTALLATION_ROOT`: docker compose -f docker-compose.yaml up ---- -Watch the output and make sure that docker containers `storage`, `server` and `db-single` are started successfully. +Watch the output and make sure that the Docker containers `storage`, `server` and `db-single` are started successfully. == Download and start the NOM agent @@ -201,27 +201,28 @@ Full documentation on registering an agent is *xref:addition/agent-installation/ * Wait for the server container to start and then go to https://server:8080. * Login as `admin` / `passw0rd` and accept license terms. * Click the top right settings icon that redirects you to the global settings. -* Make sure that the agent is online. Rename the agent if required. +* Make sure that the agent is online. +Rename the agent if required. * Return to the main page and wait for DBMS to appear - this may take a few minutes. Once the DBMS is shown in the home page, double-click on the name (initially a generated string) to edit it. Double-click on the DBMS to see the metrics, status, security panel, logs and upgrade pages for the DBMS. -== Controlling docker containers +== Controlling Docker containers === Stopping -* To stop the complete NOM environment, press `Ctrl-C` on the docker compose console and the agent console. -* To stop a single docker container, issue `docker container stop `. +* To stop the complete NOM environment, press `Ctrl-C` on the Docker compose console and the agent console. +* To stop a single Docker container, issue `docker container stop `. To list containers use `docker ps`. [NOTE] ==== -Since docker keeps persisted data in container volumes, restarted containers will keep the previous state. +Since Docker keeps persisted data in container volumes, restarted containers will keep the previous state. ==== === Resetting -To start over with an empty Neo4j persistence and empty managed instance, use the following docker command (use `docker ps -a` to find out actual container names): +To start over with an empty Neo4j persistence and empty managed instance, use the following Docker command (use `docker ps -a` to find out actual container names): [source, shell] ---- diff --git a/modules/ROOT/pages/installation/docker/compose.adoc b/modules/ROOT/pages/installation/docker/compose.adoc index a96ed877..0bd70613 100644 --- a/modules/ROOT/pages/installation/docker/compose.adoc +++ b/modules/ROOT/pages/installation/docker/compose.adoc @@ -9,10 +9,11 @@ Following is a set of steps to create a two container compose setup to experimen [NOTE] ==== -Please refer to xref:first-look/docker-first-look.adoc[Docker first look guide] for a complete example of a docker compose environment with NOM persistence, NOM server and a managed Neo4j instance. +Please refer to xref:first-look/docker-first-look.adoc[Docker first look guide] for a complete example of a Docker compose environment with NOM persistence, NOM server and a managed Neo4j instance. ==== -* Define a storage service that uses an Enterprise Neo4j image. Follow the below template and replace appropriate user defined values. +* Define a storage service that uses an Enterprise Neo4j image. +Follow the below template and replace appropriate user defined values. + [source, yaml, role=noheader] ---- diff --git a/modules/ROOT/pages/installation/docker/container.adoc b/modules/ROOT/pages/installation/docker/container.adoc index bed036ce..2056aeb9 100644 --- a/modules/ROOT/pages/installation/docker/container.adoc +++ b/modules/ROOT/pages/installation/docker/container.adoc @@ -11,7 +11,7 @@ docker pull neo4j/neo4j-ops-manager-server:latest ---- NOM server in a Docker container can be run in several contexts such as a Docker compose environment or a in a Kubernetes cluster etc. -In this chapter, we only focus on running a standalone NOM server in a Docker container (Henceforth referred as NOM server container). +This chapter focuses only on running a standalone NOM server in a Docker container (henceforth referred as NOM server container). == Running a NOM server in a Docker container diff --git a/modules/ROOT/pages/installation/server.adoc b/modules/ROOT/pages/installation/server.adoc index e80d7211..4dd8e743 100644 --- a/modules/ROOT/pages/installation/server.adoc +++ b/modules/ROOT/pages/installation/server.adoc @@ -311,7 +311,9 @@ If the NOM Server is required to support self-registered agents, then additional | `grpc.server.security.clientAuth` | `GRPC_SERVER_SECURITY_CLIENT_AUTH` -| Option to indicate whether to verify client certificates. Required for agent self-registration. Possible values: `OPTIONAL` to simultaneously allow manual registration or `REQUIRE` to only allow mutually authenticated agents. +| Option to indicate whether to verify client certificates. +Required for agent self-registration. +Possible values: `OPTIONAL` to simultaneously allow manual registration or `REQUIRE` to only allow mutually authenticated agents. | `OPTIONAL` | `logging.file.name` From 6d015c2a886a72494aca302cef5f929f9f73295c Mon Sep 17 00:00:00 2001 From: Alexander Bouriakov Date: Tue, 26 Sep 2023 13:47:48 +0200 Subject: [PATCH 14/14] updated to use 5.x Neo4j setting names --- modules/ROOT/pages/addition/docker/compose.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/pages/addition/docker/compose.adoc b/modules/ROOT/pages/addition/docker/compose.adoc index a13f1df9..a5a92246 100644 --- a/modules/ROOT/pages/addition/docker/compose.adoc +++ b/modules/ROOT/pages/addition/docker/compose.adoc @@ -43,12 +43,12 @@ test_local: NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes" NEO4J_AUTH: neo4j/passw0rd NEO4J_EDITION: "enterprise" - NEO4J_metrics_prometheus_enabled: "true" - NEO4J_metrics_prometheus_endpoint: "localhost:2004" - NEO4J_metrics_filter: "*" + NEO4J_server_metrics_prometheus_enabled: "true" + NEO4J_server_metrics_prometheus_endpoint: "localhost:2004" + NEO4J_server_metrics_filter: "*" volumes: - /path/to/custom/entrypoint:/custom # - /path/to/agent/bin:/var/lib/neo4j/bin ---- -Above Neo4j instance as a Docker compose service can be used with similar instances as part of single compose deployment in test and local environments. \ No newline at end of file +Above Neo4j instance as a Docker compose service can be used with similar instances as part of single compose deployment in test and local environments.