Document environment variables

Author: JohT

COMMANDS.md

@@ -115,7 +115,7 @@ Change into the [cypher](./cypher/) directory e.g. with `cd cypher` and then exe
 Change into the [scripts](./scripts/) directory e.g. with `cd scripts` and then execute the script [generateScriptReference.sh](./scripts/generateScriptReference.sh) with the following command:
 
 ```script
-./../scripts/generateScriptReference.sh
+./generateScriptReference.sh
 ```
 
 ### Update Markdown Reference
@@ -125,11 +125,18 @@ Change into the [results](./results/) directory e.g. with `cd results` and then
 👉**Note:** This script is automatically triggered at the end of [copyReportsIntoResults.sh](./scripts/copyReportsIntoResults.sh)
 which is included in the pipeline [code-reports.yml](.github/workflows/code-reports.yml) and doesn't need to be executed manually normally.
 
-
 ```script
 ./../scripts/generateScriptReference.sh
 ```
 
+### Update Environment Variable Reference
+
+Change into the [scripts](./scripts/) directory e.g. with `cd scripts` and then execute the script [generateEnvironmentVariablesReference.sh](./scripts/generateEnvironmentVariablesReference.sh) with the following command:
+
+```script
+./generateEnvironmentVariablesReference.sh
+```
+
 ## Manual Setup
 
 The manual setup is only documented for completeness. It isn't needed since the analysis also covers download, installation and configuration of all needed tools.

README.md

@@ -94,6 +94,10 @@ The [Code Reports Pipeline](./.github/workflows/code-reports.yml) utilizes [GitH
 
 [COMMANDS.md](./COMMANDS.md) contains further details on commands and how to do a manual setup.
 
+## 🛠 Environment Variable Reference
+
+[ENVIRONMENT_VARIABLES.md](./scripts/ENVIRONMENT_VARIABLES.md) contains all environment variables that are used in the scripts including default values and description. It can updated as described in [Update Environment Variable Reference](./COMMANDS.md#update-environment-variable-reference) of the [Commands Reference](./COMMANDS.md).
+
 ## 🤔 Questions & Answers
 
 - How can i run an analysis locally?  

scripts/ENVIRONMENT_VARIABLES.md

@@ -0,0 +1,43 @@
+# Environment Variables Reference
+
+This document serves as a reference for all environment variables that are supported by the script files.
+It provides a table listing each environment variable, its default value and its corresponding description provided as a inline comment.
+
+| Environment Variable Name           | Default                             | Description                                            |
+| ----------------------------------- | ----------------------------------- | ------------------------------------------------------ |
+REPORTS_SCRIPTS_DIRECTORY             | reports                             | Working directory containing the generated reports |
+REPORT_COMPILATIONS_SCRIPTS_DIRECTORY | compilations                        | Repository directory that contains scripts that execute selected report generation scripts |
+SETTINGS_PROFILE_SCRIPTS_DIRECTORY    | profiles                            | Repository directory that contains scripts containing settings |
+ARTIFACTS_DIRECTORY                   | artifacts                           | Working directory containing the artifacts to be analyzed |
+RESULTS_DIRECTORY                     | results                             | Repository directory containing the final analysis report results |
+REPORTS_DIRECTORY                     | reports                             | Working directory where the analysis reports are written to  |
+ARTIFACTS_CHANGE_DETECTION_HASH_FILE  | artifactsChangeDetectionHash.txt    | Name of the file that contains the hash code of the file list for change detection |
+SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION  |                                     | Skip PDF generation for Jupyter Notebooks if set to a non empty value e.g. "true" |
+JUPYTER_OUTPUT_FILE_POSTFIX           |                                     | e.g. "" (no postfix), ".nbconvert" or ".output" |
+CODEGRAPH_CONDA_ENVIRONMENT           | codegraph                           | Name of the conda environment to use for code graph analysis |
+NEO4J_HTTP_PORT                       | 7474                                | Neo4j HTTP API port for executing queries |
+NEO4J_HTTP_TRANSACTION_ENDPOINT       | db/neo4j/tx/commit                  | Neo4j v5: "db/<name>/tx/commit", Neo4j v4: "db/data/transaction/commit" |
+CYPHER_DIR                            | ${SCRIPTS_DIR}/../cypher            | Repository directory containing the cypher queries |
+NEO4J_VERSION                         | 4.4.20                              | Version 4.4.x is the current long term support (LTS) version (may 2023) |
+NEO4J_HTTPS_PORT                      | 7473                                | Neo4j HTTPS port for encrypted querying |
+NEO4J_BOLT_PORT                       | 7687                                | Neo4j's own "Bolt Protocol" port |
+NEO4J_APOC_PLUGIN_VERSION             | 4.4.0.15                            | Version number matches Neo4j version |
+NEO4J_APOC_PLUGIN_EDITION             | all                                 | Since Neo4j v5 only the core edition is maintained |
+NEO4J_APOC_PLUGIN_GITHUB              | neo4j-contrib/neo4j-apoc-procedures | Location for the old plugins compatible to Neo4j v4 |
+NEO4J_GDS_PLUGIN_VERSION              | 2.3.4                               | Graph Data Science Plugin Version 2.3.x is compatible with Neo4j 4.4.x |
+NEO4J_GDS_PLUGIN_EDITION              | full                                | Graph Data Science Plugin Edition: "open" for OpenGDS, "full" for the full version with Neo4j license |
+JQASSISTANT_CLI_VERSION               | 1.12.2                              | Version 1.12.2 is the newest version (may 2023) compatible with Neo4j v4 |
+JQASSISTANT_CLI_ARTIFACT              | jqassistant-commandline-neo4jv3     | For Neo4j v3 & 4: "jqassistant-commandline-neo4jv3" |
+JQASSISTANT_CLI_DISTRIBUTION          | distribution.zip                    | Neo4j v3 & 4: "distribution.zip" |
+JQASSISTANT_CONFIG_TEMPLATE           | template-neo4jv4-jqassistant.yaml   | Name of the template file for the jqassistant configuration |
+NEO4J_OPEN_GDS_PLUGIN_VERSION         | 2.4.3                               | Graph Data Science Plugin Version 2.4.x of is compatible with Neo4j 5.x |
+SCRIPTS_DIR                           | ${REPORTS_SCRIPT_DIR}/..            | Repository directory containing the shell scripts |
+JUPYTER_NOTEBOOK_DIRECTORY            | ${SCRIPTS_DIR}/../jupyter           | Repository directory containing the Jupyter Notebooks |
+NEO4J_EDITION                         | community                           | Choose "community" or "enterprise" |
+NEO4J_BOLT_URI                        | bolt://localhost:${NEO4J_BOLT_PORT} | Neo4j's own "Bolt Protocol" address |
+NEO4J_USER                            | neo4j                               | Neo4j login user |
+NEO4J_INITIAL_PASSWORD                |                                     | Neo4j login password that was set to replace the temporary initial password |
+TOOLS_DIRECTORY                       | tools                               | Get the tools directory (defaults to "tools") |
+JQASSISTANT_CLI_DOWNLOAD_URL          | https://repo1.maven.org/maven2/com/buschmais/jqassistant/cli | Download URL for the jQAssistant CLI |
+NEO4J_DATA_PATH                       | $( pwd -P )/data                    | Path where Neo4j writes its data to (outside tools dir) |
+NEO4J_RUNTIME_PATH                    | $( pwd -P )/runtime                 | Path where Neo4j puts runtime data to (e.g. logs) (outside tools dir) |

scripts/analysis/analyze.sh

@@ -31,10 +31,10 @@
 # Requires setupNeo4j.sh,setupJQAssistant.sh,startNeo4j.sh,resetAndScanChanged.sh,prepareAnalysis.sh,stopNeo4j.sh,comilations/*.sh,profiles/*.sh
 
 # Overrideable variables with directory names
-REPORTS_SCRIPTS_DIRECTORY=${REPORTS_SCRIPTS_DIRECTORY:-"reports"}
-REPORT_COMPILATIONS_SCRIPTS_DIRECTORY=${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY:-"compilations"}
-SETTINGS_PROFILE_SCRIPTS_DIRECTORY=${SETTINGS_PROFILE_SCRIPTS_DIRECTORY:-"profiles"}
-ARTIFACTS_DIRECTORY=${ARTIFACTS_DIRECTORY:-"artifacts"}
+REPORTS_SCRIPTS_DIRECTORY=${REPORTS_SCRIPTS_DIRECTORY:-"reports"} # Working directory containing the generated reports
+REPORT_COMPILATIONS_SCRIPTS_DIRECTORY=${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY:-"compilations"} # Repository directory that contains scripts that execute selected report generation scripts
+SETTINGS_PROFILE_SCRIPTS_DIRECTORY=${SETTINGS_PROFILE_SCRIPTS_DIRECTORY:-"profiles"} # Repository directory that contains scripts containing settings
+ARTIFACTS_DIRECTORY=${ARTIFACTS_DIRECTORY:-"artifacts"} # Working directory containing the artifacts to be analyzed
 
 # Function to display script usage
 usage() {
@@ -97,7 +97,7 @@ ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BA
 echo "analyze: ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR}"
 
 # Get the "scripts" directory by taking the path of this script and going one directory up.
-SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${ANALYSIS_SCRIPT_DIR}")}
+SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${ANALYSIS_SCRIPT_DIR}")} # Repository directory containing the shell scripts
 echo "analyze: SCRIPTS_DIR=${SCRIPTS_DIR}"
 
 # Assure that there is a report compilation script for the given report argument.

scripts/appendEnvironmentVariables.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Extracts the environment variable declarations including default values from a script file and appends it to a markdown file as table columns.
+
+# Markdown file name
+markdownFile="ENVIRONMENT_VARIABLES.md"
+
+# Get the file to search for environment variables from the first (and only) command line option
+filePath="$1"
+
+# Check if the given file exists
+if [ -z "${filePath}" ] ; then
+    echo "findEnvironmentVariables: Please provide a file name."
+    exit 0
+fi
+
+# Check if the given file exists
+if [ ! -f "./${filePath}" ] ; then
+    echo "findEnvironmentVariables: File ${filePath} doesn't exist."
+    exit 0
+fi
+
+# Create the markdown file if it doesn't exist with the header of the table
+if [ ! -f "./${markdownFile}" ] ; then
+    echo "findEnvironmentVariables: Creating ${markdownFile}..."
+    { 
+        echo "# Environment Variables Reference" 
+        echo ""
+        echo "This document serves as a reference for all environment variables that are supported by the script files."
+        echo "It provides a table listing each environment variable, its default value and its corresponding description provided as a inline comment."
+        echo ""
+        echo "| Environment Variable Name           | Default                             | Description                                            |"
+        echo "| ----------------------------------- | ----------------------------------- | ------------------------------------------------------ |"
+    } > ${markdownFile}
+fi
+
+# Regular expression that extracts the environment variable name (1st group), its default value (2nd group) and its description (3rd group)
+regular_expression='^[[:space:]]*([A-Za-z_][A-Za-z0-9_]*)=\${[A-Za-z_][A-Za-z0-9_]*:-"([^"]*)"}[[:space:]]*#*[[:space:]]*(.*)'
+
+environmentVariables=$(\
+    # Use grep to find lines with the pattern you specified
+    grep -E "${regular_expression}" "$filePath" |
+    # Use sed to extract the variable name and default value
+    sed -E "s/${regular_expression}/\1;\2;\3/" |
+    # Use awk to format the output as requested
+    awk -F ';' '{ printf "%-37s | %-35s | %s |\n", $1, $2, $3 }'\
+)
+
+# Iterate over each environment variable line by line
+while IFS= read -r environmentVariable; do
+    # Extract the name of the environment variable out of the markdown table line format.
+    environmentVariableName=$(echo -n "${environmentVariable}" | tr -s " " | cut -d "|" -f 1)
+    # Check if the environment variable is already in the output markdown file.
+    if ! grep -q "${environmentVariableName}" "$markdownFile"; then
+        # Append the previously found environment variable to the markdown file since it isn't in there yet.
+        echo "${environmentVariable}" >> ${markdownFile}
+    fi
+done <<< "$environmentVariables"

scripts/copyReportsIntoResults.sh

@@ -12,13 +12,13 @@
 # Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. 
 # CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
 # This way non-standard tools like readlink aren't needed.
-SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )}
+SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts
 echo "copyReportsIntoResults: SCRIPTS_DIR=$SCRIPTS_DIR"
 
-RESULTS_DIRECTORY=${RESULTS_DIRECTORY:-"results"}
+RESULTS_DIRECTORY=${RESULTS_DIRECTORY:-"results"} # Repository directory containing the final analysis report results
 echo "copyReportsIntoResults: RESULTS_DIRECTORY=${RESULTS_DIRECTORY}"
 
-REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"}
+REPORTS_DIRECTORY=${REPORTS_DIRECTORY:-"reports"} # Working directory where the analysis reports are written to 
 echo "copyReportsIntoResults: REPORTS_DIRECTORY=${REPORTS_DIRECTORY}"
 
 FULL_RESULTS_DIRECTORY="./../${RESULTS_DIRECTORY}"

scripts/detectChangedArtifacts.sh

@@ -5,7 +5,7 @@
 # A change is detected when the current hash and the stored one differ.
 
 ARTIFACTS_DIRECTORY=${ARTIFACTS_DIRECTORY:-"artifacts"}
-ARTIFACTS_CHANGE_DETECTION_HASH_FILE=${ARTIFACTS_CHANGE_DETECTION_HASH_FILE:-"artifactsChangeDetectionHash.txt"}
+ARTIFACTS_CHANGE_DETECTION_HASH_FILE=${ARTIFACTS_CHANGE_DETECTION_HASH_FILE:-"artifactsChangeDetectionHash.txt"} # Name of the file that contains the hash code of the file list for change detection
 ARTIFACTS_CHANGE_DETECTION_HASH_FILE_PATH="./${ARTIFACTS_DIRECTORY}/$ARTIFACTS_CHANGE_DETECTION_HASH_FILE"
 
 # Check if the artifacts directory exists

scripts/downloadMavenArtifact.sh

@@ -57,7 +57,7 @@ fi
 # Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. 
 # CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
 # This way non-standard tools like readlink aren't needed.
-SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )}
+SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts
 
 # Internal constants
 BASE_URL="https://repo1.maven.org/maven2"

scripts/downloader/downloadAxonFramework.sh

@@ -36,7 +36,7 @@ DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname --
 echo "download${ANALYSIS_NAME}: DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR}"
 
 # Get the "scripts" directory by taking the path of this script and going one directory up.
-SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${DOWNLOADER_SCRIPTS_DIR}")}
+SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${DOWNLOADER_SCRIPTS_DIR}")} # Repository directory containing the shell scripts
 echo "download${ANALYSIS_NAME}: SCRIPTS_DIR=${SCRIPTS_DIR}"
 
 ################################################################

scripts/executeJupyterNotebook.sh

@@ -21,7 +21,7 @@
 
 # Requires juypter nbconvert
 
-SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION=${SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION:-""}
+SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION=${SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION:-""} # Skip PDF generation for Jupyter Notebooks if set to a non empty value e.g. "true"
 
 # Check if environment variable is set
 if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
@@ -77,7 +77,7 @@ if [ ! -f "${jupyter_notebook_file_path}/.env" ] ; then
 fi
 
 # Define conda environment to use for code structure analysis. Default "codegraph"
-CODEGRAPH_CONDA_ENVIRONMENT=${CODEGRAPH_CONDA_ENVIRONMENT:-"codegraph"}
+CODEGRAPH_CONDA_ENVIRONMENT=${CODEGRAPH_CONDA_ENVIRONMENT:-"codegraph"} # Name of the conda environment to use for code graph analysis
 echo "executeJupyterNotebook: CODEGRAPH_CONDA_ENVIRONMENT=$CODEGRAPH_CONDA_ENVIRONMENT"
 
 # Determine the path to "conda"