Separate artifacts & their download from analysis

Author: JohT

.github/workflows/code-reports.yml

@@ -97,13 +97,21 @@ jobs:
         key:
           ${{ runner.os }}-${{ hashFiles('**/*.sh') }}
 
+    - name: Download AxonFramework artifacts
+      working-directory: temp
+      run: |
+        mkdir -p "./AxonFramework-${{ env.AXON_FRAMEWORK_VERSION }}"
+        cd "./AxonFramework-${{ env.AXON_FRAMEWORK_VERSION }}"
+        ./../../scripts/downloader/downloadAxonFramework.sh ${{ env.AXON_FRAMEWORK_VERSION }}
+
     - name: Analyze AxonFramework
       working-directory: temp
       # Shell type can be skipped if jupyter notebook reports (and therefore conda) aren't needed
       shell: bash -el {0}
-      run: ./../scripts/analysis/analyze.sh --name AxonFramework --version ${{ env.AXON_FRAMEWORK_VERSION }} --report All --profile Neo4jv5
       env:
         NEO4J_INITIAL_PASSWORD: ${{ secrets.NEO4J_INITIAL_PASSWORD }}
+      run: |
+        ./../scripts/analysis/analyze.sh --report All --profile Neo4jv5
 
     - name: Move reports from the temp to the results directory preserving their surrounding directory
       working-directory: temp

COMMANDS.md

@@ -2,28 +2,62 @@
 
 ## Start an analysis
 
-Use the following commands in the root directory of this repository to start an analysis manually e.g. for [AxonFramework](./scripts/artifacts/downloadAxonFramework.sh).
+1. Create a directory for all analysis projects
 
-```shell
-export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
-mkdir -p ./temp
-cd temp
-./../scripts/analysis/analyze.sh --name AxonFramework --version 4.8.0
-```
+    ```shell
+    mkdir temp
+    cd temp
+    ```
+
+1. Create a working directory for your specific analysis
+  
+    ```shell
+    mkdir MyFirstAnalysis
+    cd MyFirstAnalysis
+    ```
+
+1. Choose an initial password for Neo4j
+
+    ```shell
+    export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
+    ```
+
+1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards)
+
+    ```shell
+    mkdir artifacts
+    ```
 
-Add the command line argument `--report Csv` to only run the CSV reports when you don't have Python set up
-or want to skip Jupyter Notebooks.
+1. Move the artifacts you want to analyze into the `artifacts` directory
 
-Add the command line argument `--profile Neo4jv4` if you want to use the older long term support (june 2023)
-version v4.4.x of Neo4j and compatible versions of plugins and JQAssistant.
+1. Optionally run a predefined script to download artifacts
+
+    ```shell
+    ./../../scripts/downloader/downloadAxonFramework.sh <version>
+    ```
+
+1. Optionally use a script to download artifacts from Maven ([details](#download-maven-artifacts-to-analyze))
+
+1. Start the analysis
+
+    ```shell
+    ./../../scripts/analysis/analyze.sh
+    ```
+
+<span style="font-size:1.8em;">&#128214;</span>
+See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example for all the above steps. See [code-reports Pipeline](./.github/workflows/code-reports.yml) on how to do this within a GitHub Actions Workflow.
+
+<span style="font-size:1.6em;">&#9432;</span>
+Add the command line argument `--report Csv` of [analyze.sh](./scripts/analysis/analyze.sh) to only generate CSV reports. This speeds up the report generation and doesn't depend on Python, Jupyter Notebook or any other related dependencies.
+
+<span style="font-size:1.6em;">&#9432;</span>
+Add the command line argument `--profile Neo4jv4` of [analyze.sh](./scripts/analysis/analyze.sh) if you want to use the older long term support (june 2023) version v4.4.x of Neo4j and compatible versions of plugins and JQAssistant.
 
 ### Notes
 
-- Be sure to use Java 11 (Mai 2023 Neo4j v4 requirement) or Java 17 (June 2023 Neo4j v5 and jQAssistant CLI v2)
+- Be sure to use Java 17 for Neo4j v5 and Java 11 for Neo4j v4
 - Use your own initial Neo4j password
 - For more details have a look at the script [analyze.sh](./scripts/analysis/analyze.sh)
-- The script file names (without the prefix "download" and without the file extension) in the directory [scripts/artifacts](./scripts/artifacts) provide all analysis names that are available.
-
 
 Have a look at [code-reports.yml](./.github/workflows/code-reports.yml) for all details about setup steps and full automation.
 

README.md

@@ -58,10 +58,10 @@ The [Code Reports Pipeline](./.github/workflows/code-reports.yml) utilizes [GitH
 - [Checkout GIT Repository](https://github.com/actions/checkout)
 - [Setup Java](https://github.com/actions/setup-java)
 - [Setup Python with Conda](https://github.com/conda-incubator/setup-miniconda) package manager [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge)
+- Download artifacts that contain the code to be analyzed [scripts/artifacts](./scripts/downloader/)
 - Setup [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
 - Setup [jQAssistant](https://jqassistant.org/get-started) for Java Analysis ([analysis.sh](./scripts/analysis/analyze.sh))
 - Start [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
-- Trigger Artifacts download that contain the code to be analyzed [scripts/artifacts](./scripts/artifacts/)
 - Generate CSV Reports [scripts/reports](./scripts/reports) using the command line JSON parser [jq](https://jqlang.github.io/jq)
 - Generate [Jupyter Notebook](https://jupyter.org) reports using these libraries specified in the [environment.yml](./jupyter/environment.yml):
   - [Python](https://www.python.org)
@@ -97,7 +97,7 @@ The [Code Reports Pipeline](./.github/workflows/code-reports.yml) utilizes [GitH
 ## 🤔 Questions & Answers
 
 - How can i run an analysis locally?  
-  👉 See [start-an-analysis](./COMMANDS.md#start-an-analysis) in the [Commands Reference](./COMMANDS.md).
+  👉 See [Start an analysis](./COMMANDS.md#start-an-analysis) in the [Commands Reference](./COMMANDS.md).
 
 - How can i add an CSV report to the pipeline?  
   👉 Put your new cypher query into the [cypher](./cypher) directory or a suitable (new) sub directory.  
@@ -109,12 +109,12 @@ The [Code Reports Pipeline](./.github/workflows/code-reports.yml) utilizes [GitH
   👉 Create a new Jupyter report script in the [scripts/reports](./scripts/reports/) directory. Take [OverviewJupyter.sh](./scripts/reports/OverviewJupyter.sh) as a reference for example.  
   👉 The script will automatically be included because of the directory and its name ending with "Jupyter.sh".
 
-- How can i add another code base to analyze?  
-  👉 Create an new artifacts download script in the [scripts/artifacts](./scripts/artifacts) directory. Take [downloadAxonFramework.](./scripts/artifacts/downloadAxonFramework.sh) as a reference for example.  
-  👉 The script will be triggered when the [analyze](./scripts/analysis/analyze.sh) command
+- How can i add another code basis to be analyzed automatically?  
+  👉 Create a new artifacts download script in the [scripts/artifacts](./scripts/artifacts) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
+  👉 Run the script separately before executing [analyze.sh](./scripts/analysis/analyze.sh) also in the [pipeline](./.github/workflows/code-reports.yml).
 
 - How can i trigger a full rescan of all artifacts?  
-  👉 Delete the file `artifactsChangeDetectionHash.txt` in the temporary `artifacts` directory.
+  👉 Delete the file `artifactsChangeDetectionHash.txt` in the `artifacts` directory.
 
 - How can PDF generation be skipped to speed up report generation and not depend on chromium?  
   👉 Set environment variable `SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION` to anything except an empty string. Example:  

scripts/SCRIPTS.md

@@ -6,10 +6,10 @@ It provides a table listing each script file and its corresponding description f
 Script | Directory | Description
 -------|-----------|------------
 | [analyze.sh](./analysis/analyze.sh) | analysis | Coordinates the end-to-end analysis process, encompassing tool installation, graph generation, and report generation. |
-| [downloadAxonFramework.sh](./artifacts/downloadAxonFramework.sh) | artifacts | Downloads AxonFramework (https://developer.axoniq.io/axon-framework) artifacts from Maven Central. |
 | [copyReportsIntoResults.sh](./copyReportsIntoResults.sh) |  | Copies the results from the temp directory to the results directory grouped by the analysis name. |
 | [detectChangedArtifacts.sh](./detectChangedArtifacts.sh) |  | Detect changed files in the artifacts directory with a text file containing the last hash code of the contents. |
 | [downloadMavenArtifact.sh](./downloadMavenArtifact.sh) |  | Downloads an artifact from Maven Central (https://mvnrepository.com/repos/central) |
+| [downloadAxonFramework.sh](./downloader/downloadAxonFramework.sh) | downloader | Downloads AxonFramework (https://developer.axoniq.io/axon-framework) artifacts from Maven Central. |
 | [executeJupyterNotebook.sh](./executeJupyterNotebook.sh) |  | Executes all steps in the given Jupyter Notebook (ipynb), stores it and converts it to Markdown (md) and PDF. |
 | [executeQuery.sh](./executeQuery.sh) |  | Utilizes Neo4j's HTTP API to execute a Cypher query from an input file and provides the results in CSV format. |
 | [executeQueryFunctions.sh](./executeQueryFunctions.sh) |  | Provides functions to execute Cypher queries using either "executeQuery.sh" or Neo4j's "cypher-shell".  |

scripts/analysis/analyze.sh

@@ -2,21 +2,14 @@
 
 # Coordinates the end-to-end analysis process, encompassing tool installation, graph generation, and report generation.
 # - Download and setup Neo4j and JQAssistant
-# - Trigger artifacts download
-# - Scan and analyze the artifacts to create the graph 
+# - Scan and analyze the contents of the artifacts directory to create the graph 
 # - Trigger all requested reports
 
-# Note: Everything is done in the current directory. 
-#       It is recommended to create an empty directory (preferrable "temp") 
+# Note: Everything is done in the current (=working) directory and one directory above (shared downloads). 
+#       It is recommended to create an empty directory (preferrable "temp") and
+#       within that another one for the analysis (e.g. "MyCodebaseName-Version")
 #       and change into it prior to starting this script.
 
-# Note: The argument "--name" is requried. It is used to create the working directory 
-#       as well as to find the script "scripts/artifacts/download<name>.sh" to download the artifacts.
-
-# Note: The argument "--version" is also required. 
-#       It is appended to the working directory to further distinguish the results.
-#       The version is passed to the artifacts download script as an argument.
-
 # Note: The argument "--report" is optional. The default value is "All". 
 #       It selects the report compilation, a named group of reports. Besides the default "All" there are e.g. "Csv" and "Jupyter".
 #       This makes it possible to run only a subset of the reports. For example "Csv" won't need python to be set up and runs therefore much faster.
@@ -29,17 +22,16 @@
 
 # Note: The script and its sub scripts are designed to be as efficient as possible 
 #       when it comes to subsequent executions.
-#       Existing downloads, installations, artifacts, scans and processes will be detected.
+#       Existing downloads, installations, scans and processes will be detected.
 
 # Overrideable variables with directory names
-ARTIFACT_SCRIPTS_DIRECTORY=${ARTIFACT_SCRIPTS_DIRECTORY:-"artifacts"}
 REPORTS_SCRIPTS_DIRECTORY=${REPORTS_SCRIPTS_DIRECTORY:-"reports"}
 REPORT_COMPILATIONS_SCRIPTS_DIRECTORY=${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY:-"compilations"}
 SETTINGS_PROFILE_SCRIPTS_DIRECTORY=${SETTINGS_PROFILE_SCRIPTS_DIRECTORY:-"profiles"}
 
 # Function to display script usage
 usage() {
-  echo "Usage: $0 --name <name> --version <version> [--report <All (default), Csv, Jupyter,...>] [--profile <Default, Neo4jv5, Neo4jv4,...>]"
+  echo "Usage: $0 [--report <All (default), Csv, Jupyter,...>] [--profile <Default, Neo4jv5, Neo4jv4,...>]"
   exit 1
 }
 
@@ -51,14 +43,6 @@ settingsProfile="Default"
 while [[ $# -gt 0 ]]; do
   key="$1"
   case $key in
-    --name)
-      analysisName="${2}"
-      shift
-      ;;
-    --version)
-      analysisVersion="$2"
-      shift
-      ;;
     --report)
       analysisReportCompilation="$2"
       shift
@@ -75,33 +59,15 @@ while [[ $# -gt 0 ]]; do
   shift
 done
 
-# Check if the name is provided
-if [[ -z ${analysisName} ]]; then
-  echo "analyze ${analysisName}: Error: Name is required."
-  usage
-fi
-
-# Check if the version is provided
-if [[ -z ${analysisVersion} ]]; then
-  echo "analyze ${analysisName}: Error: Version is required."
-  usage
-fi
-
-# Assure that the analysis name only consists of letters and numbers
-if ! [[ ${analysisName} =~ ^[[:alnum:]]+$ ]]; then
-  echo "analyze ${analysisName}: Error: Name can only contain letters and numbers."
-  exit 1
-fi
-
 # Assure that the analysis report compilation only consists of letters and numbers
 if ! [[ ${analysisReportCompilation} =~ ^[[:alnum:]]+$ ]]; then
-  echo "analyze ${analysisName}: Report can only contain letters and numbers."
+  echo "analyze: Report can only contain letters and numbers."
   exit 1
 fi
 
 # Assure that the settings profile only consists of letters and numbers
 if ! [[ ${settingsProfile} =~ ^[[:alnum:]]+$ ]]; then
-  echo "analyze ${analysisName}: Error: Settings profile can only contain letters and numbers."
+  echo "analyze: Error: Settings profile can only contain letters and numbers."
   exit 1
 fi
 
@@ -110,66 +76,46 @@ fi
 # CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
 # This way non-standard tools like readlink aren't needed.
 ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )}
-echo "analyze ${analysisName}: ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR}"
+echo "analyze: ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR}"
 
 # Get the "scripts" directory by taking the path of this script and going one directory up.
 SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${ANALYSIS_SCRIPT_DIR}")}
-echo "analyze ${analysisName}: SCRIPTS_DIR=${SCRIPTS_DIR}"
-
-# Assure that there is a download script for the given analysis name argument.
-DOWNLOAD_SCRIPT="${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY}/download${analysisName}.sh"
-if [ ! -f "${DOWNLOAD_SCRIPT}" ] ; then
-  echo "analyze ${analysisName}: Error: No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
-  exit 1
-fi
+echo "analyze: SCRIPTS_DIR=${SCRIPTS_DIR}"
 
 # Assure that there is a report compilation script for the given report argument.
 REPORT_COMPILATION_SCRIPT="${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY}/${analysisReportCompilation}Reports.sh"
 if [ ! -f "${REPORT_COMPILATION_SCRIPT}" ] ; then
-  echo "analyze ${analysisName}: Error: No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
+  echo "analyze: Error: No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
   exit 1
 fi
 
 # Assure that there is a script file for the given settings profile name.
 SETTINGS_PROFILE_SCRIPT="${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY}/${settingsProfile}.sh"
 if [ ! -f "${SETTINGS_PROFILE_SCRIPT}" ] ; then
-  echo "analyze ${analysisName}: Error: No ${settingsProfile}.sh script in the directory ${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY} for settings profile ${settingsProfile}."
+  echo "analyze: Error: No ${settingsProfile}.sh script in the directory ${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY} for settings profile ${settingsProfile}."
   exit 1
 fi
 
 # Execute the settings profile script that sets all the neccessary settings variables (overrideable by environment variables).
-echo "analyze ${analysisName}: Using analysis settings profile script ${SETTINGS_PROFILE_SCRIPT}"
-source "${SETTINGS_PROFILE_SCRIPT}" || exit 1
-
-# Create working directory if it hadn't been created yet
-workingDirectory="${analysisName}-${analysisVersion}"
-mkdir -p "${workingDirectory}" || exit 2 # Create the working directory only if it doesn't exist
-pushd "${workingDirectory}" || exit 2 # Change into the working directory and remember the previous directory
-echo "analyze ${analysisName}: Working Directory: $( pwd -P )"
+echo "analyze: Using analysis settings profile script ${SETTINGS_PROFILE_SCRIPT}"
+source "${SETTINGS_PROFILE_SCRIPT}" || exit 2
 
 # Setup Tools
 source "${SCRIPTS_DIR}/setupNeo4j.sh" || exit 3
 source "${SCRIPTS_DIR}/setupJQAssistant.sh" || exit 3
 source "${SCRIPTS_DIR}/startNeo4j.sh" || exit 3
 
-# Execute the script "scripts/artifacts/download<analysisName" to download the artifacts that should be analyzed
-echo "Downloading artifacts with ${DOWNLOAD_SCRIPT} ..."
-source "${DOWNLOAD_SCRIPT}" "${analysisVersion}" || exit 4
-
 # Scan and analyze artifacts when they were changed
-source "${SCRIPTS_DIR}/resetAndScanChanged.sh" || exit 5
+source "${SCRIPTS_DIR}/resetAndScanChanged.sh" || exit 4
 
 # Prepare and validate graph database before analyzing and creating reports 
-source "${SCRIPTS_DIR}/prepareAnalysis.sh" || exit 6
+source "${SCRIPTS_DIR}/prepareAnalysis.sh" || exit 5
 
 #########################
 # Create Reports
 #########################
 echo "Creating Reports with ${REPORT_COMPILATION_SCRIPT} ..."
-source "${REPORT_COMPILATION_SCRIPT}" || exit 7
+source "${REPORT_COMPILATION_SCRIPT}" || exit 6
 
 # Stop Neo4j at the end
-source "${SCRIPTS_DIR}/stopNeo4j.sh"
-
-# Change back to the previous directory where the script was started
-popd || exit 8
+source "${SCRIPTS_DIR}/stopNeo4j.sh"

scripts/artifacts/downloadAxonFramework.sh renamed to scripts/downloader/downloadAxonFramework.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# This is an example for an analysis of AxonFramework 
+# including the creation of the temporary directory, the working directory, the artifacts download and the analysis itself.
+
+# Note: The first (and only) parameter is the version of AxonFramework to analyze.
+# Note: This script is meant to be started in the root directory of this repository.
+
+# Read the first input argument containing the version of the artifacts
+if [ "$#" -ne 1 ]; then
+  echo "analyzeAxonFramework Error: Usage: $0 <version>" >&2
+  exit 1
+fi
+artifactsVersion=$1
+
+# Check if environment variable is set
+if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
+    echo "analyzeAxonFramework: Error: Requires environment variable NEO4J_INITIAL_PASSWORD to be set first. Use 'export NEO4J_INITIAL_PASSWORD=<your-own-password>'."
+    exit 1
+fi
+
+# Create the temporary directory for all analysis projects.
+mkdir -p ./temp || exit 1
+cd ./temp || exit 1
+
+# Create the working directory for this specific analysis.
+mkdir -p "./AxonFramework-${artifactsVersion}" || exit 2
+cd "./AxonFramework-${artifactsVersion}"  || exit 2
+
+# Create the artifacts directory that will contain the code to be analyzed.
+mkdir -p ./artifacts
+
+# Download AxonFramework artifacts (jar files) from Maven
+./../../scripts/downloader/downloadAxonFramework.sh "${artifactsVersion}"
+
+# Start the analysis
+./../../scripts/analysis/analyze.sh