Merge pull request #6 from JohT/feature/support-neo4j-v5

Support Neo4j v5 and jQAssistant CLI 2.0

Author: JohT

.github/workflows/code-reports.yml

@@ -42,7 +42,7 @@ jobs:
       matrix:
         include:
         - os: ubuntu-latest
-          java: 11
+          java: 17
           python: 3.11
           mambaforge: 23.1.0-1
 

COMMANDS.md

@@ -4,20 +4,27 @@
 
 Use the following commands in the root directory of this repository to start an analysis manually e.g. for [AxonFramework](./scripts/artifacts/downloadAxonFramework.sh).
 
-### Notes
-
-- Be sure to use Java 11 (June 2023 Neo4j 4.x requirement)
-- Use your own initial Neo4j password
-- It uses the script [analyze.sh](./scripts/analysis/analyze.sh)
-- The script file names (without the prefix "download" and without the file extension) in the directory [scripts/artifacts](./scripts/artifacts) provide the possible analysis names.
-
-
 ```shell
 export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
 mkdir -p ./temp
-./../scripts/analysis/analyze.sh --name AxonFramework --version 4.7.5 --report All
+cd temp
+./../scripts/analysis/analyze.sh --name AxonFramework --version 4.8.0
 ```
 
+Add the command line argument `--report Csv` to only run the CSV reports when you don't have Python set up
+or want to skip Jupyter Notebooks.
+
+Add the command line argument `--profile Neo4jv4` if you want to use the older long term support (june 2023)
+version v4.4.x of Neo4j and compatible versions of plugins and JQAssistant.
+
+### Notes
+
+- Be sure to use Java 11 (Mai 2023 Neo4j v4 requirement) or Java 17 (June 2023 Neo4j v5 and jQAssistant CLI v2)
+- Use your own initial Neo4j password
+- For more details have a look at the script [analyze.sh](./scripts/analysis/analyze.sh)
+- The script file names (without the prefix "download" and without the file extension) in the directory [scripts/artifacts](./scripts/artifacts) provide all analysis names that are available.
+
+
 Have a look at [code-reports.yml](./.github/workflows/code-reports.yml) for all details about setup steps and full automation.
 
 ## Generate Markdown References

JQAssistantGraphDatabaseCompatibilityIssues.md

@@ -97,6 +97,9 @@ The [Code Reports Pipeline](./.github/workflows/code-reports.yml) utilizes [GitH
 
 ## 🤔 Questions & Answers
 
+- How can i run an analysis locally?  
+👉 See [start-an-analysis](./COMMANDS.md#start-an-analysis) in the [Commands Reference](./COMMANDS.md).
+
 - How can i add an CSV report to the pipeline?  
 👉 Put your new cypher query into the [cypher](./cypher) directory or a suitable (new) sub directory.  
 👉 Create a new CSV report script in the [scripts/reports](./scripts/reports/) directory. Take for example [OverviewCsv.sh](./scripts/reports/OverviewCsv.sh) as a reference.

README.md

@@ -0,0 +1,5 @@
+// Count nodes and relationships
+
+MATCH (n)-[r]-(m) 
+RETURN COUNT(DISTINCT n) AS nodeCount
+      ,COUNT(DISTINCT r) AS relationshipCount

cypher/Count_nodes_and_relationships.cypher

@@ -0,0 +1,7 @@
+// Create a DEPENDS_ON relationship for every DEPENDS_ON_ARTIFACT
+
+MATCH (a:Artifact)-[existing:DEPENDS_ON_ARTIFACT]->(b:Artifact)
+MERGE (a)-[created:DEPENDS_ON]->(b)
+ WITH count(existing) as numberOfExistingRelations
+     ,count(created)  as numberOfCreatedRelations
+RETURN numberOfExistingRelations, numberOfCreatedRelations

cypher/Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher

@@ -0,0 +1,7 @@
+// Create a DEPENDS_ON relationship for every DEPENDS_ON_PACKAGE
+
+MATCH (a:Package)-[existing:DEPENDS_ON_PACKAGE]->(b:Package)
+MERGE (a)-[created:DEPENDS_ON]->(b)
+ WITH count(existing) as numberOfExistingRelations
+     ,count(created)  as numberOfCreatedRelations
+RETURN numberOfExistingRelations, numberOfCreatedRelations

cypher/Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher

@@ -16,8 +16,13 @@ Script | Directory | Description
 | [generateCypherReference.sh](./generateCypherReference.sh) |  | Generates "CYPHER.md" containing a reference to all Cypher files in this directory and its subdirectories. |
 | [generateMarkdownReference.sh](./generateMarkdownReference.sh) |  | Generates "REPORTS.md" containing a reference to all scripts in this directory and its subdirectories. |
 | [generateScriptReference.sh](./generateScriptReference.sh) |  | Generates "SCRIPTS.md" containing a reference to all scripts in this directory and its subdirectories. |
+| [prepareAnalysis.sh](./prepareAnalysis.sh) |  | Prepares and validates the graph database before analysis  |
+| [Default.sh](./profiles/Default.sh) | profiles | Sets (if any) settings variables for a default analysis. |
+| [Neo4jv4.sh](./profiles/Neo4jv4.sh) | profiles | Sets all settings variables for an analysis with Neo4j v4.4.x (long term support (LTS) version as of may 2023). |
+| [Neo4jv5.sh](./profiles/Neo4jv5.sh) | profiles | Sets all settings variables for an analysis with Neo4j v5.x (newest version as of june 2023). |
 | [CentralityCsv.sh](./reports/CentralityCsv.sh) | reports | Looks for centrality using the Graph Data Science Library of Neo4j and creates CSV reports. |
 | [CommunityCsv.sh](./reports/CommunityCsv.sh) | reports | Detects communities using the Graph Data Science Library of Neo4j and creates CSV reports. |
+| [DatabaseCsvExport.sh](./reports/DatabaseCsvExport.sh) | reports | Exports the whole graph database as a CSV file using the APOC procedure "apoc.export.csv.all" |
 | [ExternalDependenciesCsv.sh](./reports/ExternalDependenciesCsv.sh) | reports | Executes "Package_Usage" Cypher queries to get the "package-dependencies" CSV reports. |
 | [ExternalDependenciesJupyter.sh](./reports/ExternalDependenciesJupyter.sh) | reports | Creates the "overview" report (ipynb, md, pdf) based on the Jupyter Notebook "Overview.ipynb". |
 | [ObjectOrientedDesignMetricsCsv.sh](./reports/ObjectOrientedDesignMetricsCsv.sh) | reports | Executes "Metrics" Cypher queries to get the "object-oriented-design-metrics" CSV reports. |
@@ -35,6 +40,7 @@ Script | Directory | Description
 | [JupyterReports.sh](./reports/compilations/JupyterReports.sh) | compilations | Runs all Jupyter Notebook report scripts. |
 | [resetAndScan.sh](./resetAndScan.sh) |  | Deletes all data in the Neo4j graph database and rescans the downloaded artifacts to create a new graph. |
 | [resetAndScanChanged.sh](./resetAndScanChanged.sh) |  | Executes "resetAndScan.sh" only if "detectChangedArtifacts.sh" returns detected changes. |
+| [resetAndScanMemgraph.sh](./resetAndScanMemgraph.sh) |  | Deletes all data in the Neo4j graph database and rescans the downloaded artifacts to create a new graph. |
 | [setupJQAssistant.sh](./setupJQAssistant.sh) |  | Installs (download and unzip) jQAssistant (https://jqassistant.org/get-started). |
 | [setupNeo4j.sh](./setupNeo4j.sh) |  | Installs (download, unpack, get plugins, configure) a local Neo4j Graph Database (https://neo4j.com/download-center/#community). |
 | [setupNeo4jInitialPassword.sh](./setupNeo4jInitialPassword.sh) |  | Sets the initial password for the local Neo4j Graph Database (https://neo4j.com/download-center/#community). |

scripts/SCRIPTS.md

@@ -10,39 +10,42 @@
 #       It is recommended to create an empty directory (preferrable "temp") 
 #       and change into it prior to starting this script.
 
-# Note: The first argument "--name" is requried. It is used to create the working directory 
+# Note: The argument "--name" is requried. It is used to create the working directory 
 #       as well as to find the script "scripts/artifacts/download<name>.sh" to download the artifacts.
 
-# Note: The second argument "--version" is optional. 
+# Note: The argument "--version" is also required. 
 #       It is appended to the working directory to further distinguish the results.
+#       The version is passed to the artifacts download script as an argument.
+
+# Note: The argument "--report" is optional. The default value is "All". 
+#       It selects the report compilation, a named group of reports. Besides the default "All" there are e.g. "Csv" and "Jupyter".
+#       This makes it possible to run only a subset of the reports. For example "Csv" won't need python to be set up and runs therefore much faster.
+#       Implemented is this as a script in "scripts/reports/compilations" that starts with the report compilation name followed by "Reports.sh".
+
+# Note: The argument "--profile" is optional. The default value is "Default". 
+#       It selects a settings profile that sets all suitable variables for the analysis.
+#       This makes it possible to run an analysis with e.g. Neo4j v4 instead of v5. Further profiles might come in future.
+#       Implemented is this as a script in "scripts/profiles" that starts with the settings profile name followed by ".sh".
 
 # Note: The script and its sub scripts are designed to be as efficient as possible 
 #       when it comes to subsequent executions.
 #       Existing downloads, installations, artifacts, scans and processes will be detected.
 
-# Overrideable environment variables (optional, defaults also defined in sub scripts where needed)
-NEO4J_EDITION=${NEO4J_EDITION:-"community"} # Choose "community" or "enterprise"
-NEO4J_VERSION=${NEO4J_VERSION:-"4.4.20"} # Version 4.4.x is the current long term support (LTS) version (may 2023)
-JQASSISTANT_CLI_VERSION=${JQASSISTANT_CLI_VERSION:-"1.12.2"} # Version 1.12.2 is the current version (may 2023)
-
-# Overrideable environment variables for ports (optional, defaults also defined in sub scripts where needed)
-# Override them if you need to run multiple neo4j database servers in parallel.
-NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"} 
-NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"}
-NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"}
-
+# Overrideable variables with directory names
 ARTIFACT_SCRIPTS_DIRECTORY=${ARTIFACT_SCRIPTS_DIRECTORY:-"artifacts"}
 REPORTS_SCRIPTS_DIRECTORY=${REPORTS_SCRIPTS_DIRECTORY:-"reports"}
 REPORT_COMPILATIONS_SCRIPTS_DIRECTORY=${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY:-"compilations"}
+SETTINGS_PROFILE_SCRIPTS_DIRECTORY=${SETTINGS_PROFILE_SCRIPTS_DIRECTORY:-"profiles"}
 
 # Function to display script usage
 usage() {
-  echo "Usage: $0 --name <name> --version <version> [--report <All (default), Csv, Jupyter,...>]"
+  echo "Usage: $0 --name <name> --version <version> [--report <All (default), Csv, Jupyter,...>] [--profile <Default, Neo4jv5, Neo4jv4,...>]"
   exit 1
 }
 
 # Default values
 analysisReportCompilation="All"
+settingsProfile="Default"
 
 # Parse command line arguments
 while [[ $# -gt 0 ]]; do
@@ -60,8 +63,12 @@ while [[ $# -gt 0 ]]; do
       analysisReportCompilation="$2"
       shift
       ;;
+    --profile)
+      settingsProfile="$2"
+      shift
+      ;;
     *)
-      echo "Error (analyze): Unknown option: ${key}"
+      echo "analyze: Error: Unknown option: ${key}"
       usage
       ;;
   esac
@@ -70,25 +77,31 @@ done
 
 # Check if the name is provided
 if [[ -z ${analysisName} ]]; then
-  echo "Error (analyze): Name is required."
+  echo "analyze ${analysisName}: Error: Name is required."
   usage
 fi
 
 # Check if the version is provided
 if [[ -z ${analysisVersion} ]]; then
-  echo "Error (analyze): Version is required."
+  echo "analyze ${analysisName}: Error: Version is required."
   usage
 fi
 
 # Assure that the analysis name only consists of letters and numbers
 if ! [[ ${analysisName} =~ ^[[:alnum:]]+$ ]]; then
-  echo "Error (analyze): Name can only contain letters and numbers."
+  echo "analyze ${analysisName}: Error: Name can only contain letters and numbers."
   exit 1
 fi
 
 # Assure that the analysis report compilation only consists of letters and numbers
 if ! [[ ${analysisReportCompilation} =~ ^[[:alnum:]]+$ ]]; then
-  echo "Error (analyze): Report can only contain letters and numbers."
+  echo "analyze ${analysisName}: Report can only contain letters and numbers."
+  exit 1
+fi
+
+# Assure that the settings profile only consists of letters and numbers
+if ! [[ ${settingsProfile} =~ ^[[:alnum:]]+$ ]]; then
+  echo "analyze ${analysisName}: Error: Settings profile can only contain letters and numbers."
   exit 1
 fi
 
@@ -103,26 +116,31 @@ echo "analyze ${analysisName}: ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR}"
 SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${ANALYSIS_SCRIPT_DIR}")}
 echo "analyze ${analysisName}: SCRIPTS_DIR=${SCRIPTS_DIR}"
 
-# Assure that there is a download script for the given analysis name.
-if [ ! -f "${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY}/download${analysisName}.sh" ] ; then
-  echo "Error (analyze): No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
-  exit 1
-fi
-
 # Assure that there is a download script for the given analysis name argument.
 DOWNLOAD_SCRIPT="${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY}/download${analysisName}.sh"
 if [ ! -f "${DOWNLOAD_SCRIPT}" ] ; then
-  echo "Error (analyze): No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
+  echo "analyze ${analysisName}: Error: No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
   exit 1
 fi
 
 # Assure that there is a report compilation script for the given report argument.
 REPORT_COMPILATION_SCRIPT="${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY}/${analysisReportCompilation}Reports.sh"
 if [ ! -f "${REPORT_COMPILATION_SCRIPT}" ] ; then
-  echo "Error (analyze): No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
+  echo "analyze ${analysisName}: Error: No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
+  exit 1
+fi
+
+# Assure that there is a script file for the given settings profile name.
+SETTINGS_PROFILE_SCRIPT="${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY}/${settingsProfile}.sh"
+if [ ! -f "${SETTINGS_PROFILE_SCRIPT}" ] ; then
+  echo "analyze ${analysisName}: Error: No ${settingsProfile}.sh script in the directory ${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY} for settings profile ${settingsProfile}."
   exit 1
 fi
 
+# Execute the settings profile script that sets all the neccessary settings variables (overrideable by environment variables).
+echo "analyze ${analysisName}: Using analysis settings profile script ${SETTINGS_PROFILE_SCRIPT}"
+source "${SETTINGS_PROFILE_SCRIPT}" || exit 1
+
 # Create working directory if it hadn't been created yet
 workingDirectory="${analysisName}-${analysisVersion}"
 mkdir -p "${workingDirectory}" || exit 2 # Create the working directory only if it doesn't exist
@@ -141,14 +159,17 @@ source "${DOWNLOAD_SCRIPT}" "${analysisVersion}" || exit 4
 # Scan and analyze artifacts when they were changed
 source "${SCRIPTS_DIR}/resetAndScanChanged.sh" || exit 5
 
+# Prepare and validate graph database before analyzing and creating reports 
+source "${SCRIPTS_DIR}/prepareAnalysis.sh" || exit 6
+
 #########################
 # Create Reports
 #########################
 echo "Creating Reports with ${REPORT_COMPILATION_SCRIPT} ..."
-source "${REPORT_COMPILATION_SCRIPT}" || exit 6
+source "${REPORT_COMPILATION_SCRIPT}" || exit 7
 
 # Stop Neo4j at the end
 source "${SCRIPTS_DIR}/stopNeo4j.sh"
 
 # Change back to the previous directory where the script was started
-popd || exit 7
+popd || exit 8

scripts/analysis/analyze.sh

@@ -15,6 +15,7 @@
 
 # Overrideable Defaults
 NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"}
+NEO4J_HTTP_TRANSACTION_ENDPOINT=${NEO4J_HTTP_TRANSACTION_ENDPOINT:-"db/neo4j/tx/commit"} # Neo4j v5: "db/<name>/tx/commit", Neo4j v4: "db/data/transaction/commit",
 
 # Check if environment variable is set
 if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
@@ -85,7 +86,7 @@ cypher_query_for_api="{\"statements\":[{\"statement\":${cypher_query},\"includeS
 # Calls the Neo4j HTTP API using cURL ( https://curl.se )
 cyper_query_result=$(curl --silent -S --fail-with-body -H Accept:application/json -H Content-Type:application/json \
      -u neo4j:"${NEO4J_INITIAL_PASSWORD}" \
-     "http://localhost:${NEO4J_HTTP_PORT}/db/data/transaction/commit" \
+     "http://localhost:${NEO4J_HTTP_PORT}/${NEO4J_HTTP_TRANSACTION_ENDPOINT}" \
      -d "${cypher_query_for_api}")
 #echo "Cypher Query Result: $cyper_query_result"