Support executing Jupyter Notebooks on Windows

Author: JohT

COMMANDS.md

@@ -2,47 +2,11 @@
 
 ## Start an analysis
 
-1. Create a directory for all analysis projects
-
-    ```shell
-    mkdir temp
-    cd temp
-    ```
-
-1. Create a working directory for your specific analysis
-  
-    ```shell
-    mkdir MyFirstAnalysis
-    cd MyFirstAnalysis
-    ```
-
-1. Choose an initial password for Neo4j
-
-    ```shell
-    export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
-    ```
-
-1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards)
-
-    ```shell
-    mkdir artifacts
-    ```
-
-1. Move the artifacts you want to analyze into the `artifacts` directory
-
-1. Optionally run a predefined script to download artifacts
-
-    ```shell
-    ./../../scripts/downloader/downloadAxonFramework.sh <version>
-    ```
-
-1. Optionally use a script to download artifacts from Maven ([details](#download-maven-artifacts-to-analyze))
-
-1. Start the analysis
-
-    ```shell
-    ./../../scripts/analysis/analyze.sh
-    ```
+An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh).
+To run all analysis steps simple execute the following command:
+```shell
+./../../scripts/analysis/analyze.sh
+```
 
 👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
 👉 See [Code Structure Analysis Pipeline](./.github/workflows/code-structure-analysis.yml) on how to do this within a GitHub Actions Workflow.

GETTING_STARTED.md

@@ -0,0 +1,73 @@
+# Code Graph Analysis Pipeline - Getting started guide
+
+This document describes the steps to get started as quickly as possible. 
+For more details on what you can do with this pipeline see [README](./README.md).
+For more details on how the commands work in detail see [COMMANDS](./COMMANDS.md).
+
+## 🛠 Prerequisites
+
+Please read through the [Prerequisites](./README.md#🛠-prerequisites) in the [README](./README.md) file for what is required to run the scripts.
+
+## Start an analysis
+
+1. Create a directory for all analysis projects
+
+    ```shell
+    mkdir temp
+    cd temp
+    ```
+
+1. Create a working directory for your specific analysis
+  
+    ```shell
+    mkdir MyFirstAnalysis
+    cd MyFirstAnalysis
+    ```
+
+1. Choose an initial password for Neo4j
+
+    ```shell
+    export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
+    ```
+
+1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards)
+
+    ```shell
+    mkdir artifacts
+    ```
+
+1. Move the artifacts you want to analyze into the `artifacts` directory
+
+1. Optionally run a predefined script to download artifacts
+
+    ```shell
+    ./../../scripts/downloader/downloadAxonFramework.sh <version>
+    ```
+
+1. Optionally use a script to download artifacts from Maven ([details](#download-maven-artifacts-to-analyze))
+
+1. Start the analysis
+
+  - Without any additional dependencies:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Csv
+    ```
+  - Jupyter notebook reports when Python and Conda are installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Jupyter
+    ```
+  - Graph visualizations when Node.js and npm are installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Jupyter
+    ```
+  - All reports with Python, Conda, Node.js and npm installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh
+    ```
+  - To explore the database yourself without any automatically generated reports and no additional requirements:
+    ```shell
+    ./../../scripts/analysis/analyze.sh
+    ```
+
+👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
+👉 See [Code Structure Analysis Pipeline](./.github/workflows/code-structure-analysis.yml) on how to do this within a GitHub Actions Workflow.

README.md

@@ -6,7 +6,7 @@ Contained within this repository is a comprehensive and automated code graph ana
 
 ---
 
-## 🚀 Features
+## ✨ Features
 
 - Analyze static code structure as a graph
 - Fully automated [pipeline](./.github/workflows/code-structure-analysis.yml) from tool installation to report generation
@@ -37,14 +37,29 @@ Here are some reports that utilize Neo4j's [Graph Data Science Library](https://
 
 ## 🛠 Prerequisites
 
-- Java 17 is [required for Neo4j](https://neo4j.com/docs/operations-manual/current/installation/requirements/#deployment-requirements-software) (5.x requirement)
+- Java 17 is [required for Neo4j](https://neo4j.com/docs/operations-manual/current/installation/requirements/#deployment-requirements-software) (Neo4j 5.x requirement).
+- On Windows it is recommended to use the git bash provided by [git for windows](https://gitforwindows.org).
 - [jq](https://github.com/jqlang/jq) the "lightweight and flexible command-line JSON processor" needs to be installed. Latest releases: https://github.com/jqlang/jq/releases/latest. Check using `jq --version`.
-- Python and a conda package manager are required for Jupyter Notebook reports
-- Chromium will automatically be downloaded if needed for Jupyter Notebook reports in PDF format
+- Set environment variable `NEO4J_INITIAL_PASSWORD` to a password of your choice. For example:
+  ```shell
+  export NEO4J_INITIAL_PASSWORD=neo4j_password_of_my_choice
+  ```
+  To run Jupyter notebooks, create an `.env` file in the folder from where you open the notebook containing for example: `NEO4J_INITIAL_PASSWORD=neo4j_password_of_my_choice`
+
+### Further Prerequisites for Python Jupyter Notebooks
+
+- Python is required for Jupyter Notebook reports.
+- A conda package manager like [Miniconda](https://docs.conda.io/projects/miniconda/en/latest) or [Anaconda](https://www.anaconda.com/download)(Recommended for Windows) is required for Jupyter Notebook reports.
+- Chromium will automatically be downloaded if needed for Jupyter Notebook PDF reports generation.
+
+### Hints for Windows
+
+- Add this line to your `~/.bashrc` file if you are using Anaconda3: `/c/ProgramData/Anaconda3/etc/profile.d/conda.sh`. Try to find a similar script for other conda package managers or versions.
+- Run `conda init` in the git bash opened as administrator. Running it in normal mode usually leads to an error message.
 
-## Getting Started
+## 🚀 Getting Started
 
-See [Start an analysis](./COMMANDS.md#start-an-analysis) in the [Commands Reference](./COMMANDS.md) on how to start an analysis on your local machine.
+See [GETTING_STARTED.md](./GETTING_STARTED.md) on how to get started on your local machine.
 
 ## 🏗 Pipeline and Tools
 

scripts/executeJupyterNotebook.sh

@@ -19,13 +19,20 @@
 #       It it isn't activated, it will save the currently activated environment, change to codegraph, and restore the original one at the end again.
 #       In cases of an error it might be, that the original conda environment isn't set back. Typically this shouldn't be the case though. 
 
-# Requires juypter nbconvert
+# Requires juypter nbconvert,,operatingSystemFunctions.sh
 
 # Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands)
 set -eo pipefail
 
 ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION=${ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION:-""} # Enable PDF generation for Jupyter Notebooks if set to any non empty value e.g. "true"
 
+## Get this "scripts" directory if not already set
+# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. 
+# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
+# This way non-standard tools like readlink aren't needed.
+SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts
+echo "startNeo4j: SCRIPTS_DIR=$SCRIPTS_DIR"
+
 # Check if environment variable is set
 if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
     echo "executeJupyterNotebook: Requires environment variable NEO4J_INITIAL_PASSWORD to be set first. Use 'export NEO4J_INITIAL_PASSWORD=<your-own-password>'."
@@ -81,36 +88,50 @@ fi
 
 # Define conda environment to use for code structure analysis. Default "codegraph"
 CODEGRAPH_CONDA_ENVIRONMENT=${CODEGRAPH_CONDA_ENVIRONMENT:-"codegraph"} # Name of the conda environment to use for code graph analysis
-echo "executeJupyterNotebook: CODEGRAPH_CONDA_ENVIRONMENT=$CODEGRAPH_CONDA_ENVIRONMENT"
+backupCondaEnvironment=$CONDA_DEFAULT_ENV
+echo "executeJupyterNotebook: CODEGRAPH_CONDA_ENVIRONMENT=${CODEGRAPH_CONDA_ENVIRONMENT}"
+echo "executeJupyterNotebook: Current conda environment=${CONDA_DEFAULT_ENV}"
+
+# Include operation system function to for example detect Windows.
+source "${SCRIPTS_DIR}/operatingSystemFunctions.sh"
+
+scriptExtension=$(ifWindows ".bat" "")
 
 # Determine the path to "conda"
 if [ -n "${CONDA}" ]; then
-    pathToConda="$CONDA/bin/" # the trailing slash character is required
+    if isWindows; then
+        pathToConda="${CONDA}\\Scripts\\" # the trailing backslash character is required
+    else
+        pathToConda="${CONDA}/bin/" # the trailing slash character is required
+    fi
 else
     pathToConda=""
 fi
 echo "executeJupyterNotebook: pathToConda=${pathToConda}"
 
-# Activate conda shell hook
-eval "$(${pathToConda}conda shell.bash hook)"
+# Activate conda shell hook. Also resets CONDA_DEFAULT_ENV to base. 
+# Thats why CONDA_DEFAULT_ENV (base) is never equal to CODEGRAPH_CONDA_ENVIRONMENT (codegraph).
+eval "$(${pathToConda}conda${scriptExtension} shell.bash hook)"
 
 # Create (if missing) and activate Conda environment for code structure graph analysis
-backupCondaEnvironment=$CONDA_DEFAULT_ENV
-if [ ! "$backupCondaEnvironment" = "$CODEGRAPH_CONDA_ENVIRONMENT" ] ; then
-    backupCondaEnvironment=$CONDA_DEFAULT_ENV
-
-    if { ${pathToConda}conda env list | grep "$CODEGRAPH_CONDA_ENVIRONMENT "; } >/dev/null 2>&1; then
-        echo "executeJupyterNotebook: Conda environment $CODEGRAPH_CONDA_ENVIRONMENT already created"
-    else
-        if [ ! -f "${jupyter_notebook_file_path}/environment.yml" ] ; then
-            echo "executeJupyterNotebook: Couldn't find environment file ${jupyter_notebook_file_path}/environment.yml."
-            exit 2
-        fi
-        echo "executeJupyterNotebook: Creating Conda environment ${CODEGRAPH_CONDA_ENVIRONMENT}"
-        ${pathToConda}conda env create --file ${jupyter_notebook_file_path}/environment.yml --name "${CODEGRAPH_CONDA_ENVIRONMENT}"
+if { "${pathToConda}conda" env list | grep "$CODEGRAPH_CONDA_ENVIRONMENT "; } >/dev/null 2>&1; then
+    echo "executeJupyterNotebook: Conda environment $CODEGRAPH_CONDA_ENVIRONMENT already created"
+else
+    if [ ! -f "${jupyter_notebook_file_path}/environment.yml" ] ; then
+        echo "executeJupyterNotebook: Couldn't find environment file ${jupyter_notebook_file_path}/environment.yml."
+        exit 2
     fi
-    ${pathToConda}conda activate "${CODEGRAPH_CONDA_ENVIRONMENT}" 
-    echo "executeJupyterNotebook: Activated Conda environment: $CODEGRAPH_CONDA_ENVIRONMENT "
+    echo "executeJupyterNotebook: Creating Conda environment ${CODEGRAPH_CONDA_ENVIRONMENT}"
+    "${pathToConda}conda" env create --file "${jupyter_notebook_file_path}/environment.yml" --name "${CODEGRAPH_CONDA_ENVIRONMENT}"
+fi
+echo "executeJupyterNotebook: Activating Conda environment ${CODEGRAPH_CONDA_ENVIRONMENT}"
+"${pathToConda}conda" activate ${CODEGRAPH_CONDA_ENVIRONMENT}
+
+if [ "${CONDA_DEFAULT_ENV}" = "${CODEGRAPH_CONDA_ENVIRONMENT}" ] ; then
+    echo "executeJupyterNotebook: Activated Conda environment: ${CONDA_DEFAULT_ENV}"
+else
+    echo "executeJupyterNotebook: Error: Failed to activate Conda environment ${CODEGRAPH_CONDA_ENVIRONMENT}. ${CONDA_DEFAULT_ENV} still active."
+    exit 1
 fi
 
 # Execute the Jupyter Notebook and write it to the output file name
@@ -135,7 +156,7 @@ if [ -n "${ENABLE_JUPYTER_NOTEBOOK_PDF_GENERATION}" ]; then
 fi
 
 # Restore Conda environment
-if [ ! "$backupCondaEnvironment" = "$CODEGRAPH_CONDA_ENVIRONMENT" ] ; then
-    ${pathToConda}conda activate "${backupCondaEnvironment}"
-    echo "executeJupyterNotebook: Restored Conda Environment: ${backupCondaEnvironment}"
+if [ ! "${backupCondaEnvironment}" = "${CODEGRAPH_CONDA_ENVIRONMENT}" ] ; then
+    "${pathToConda}conda" activate "${backupCondaEnvironment}"
+    echo "executeJupyterNotebook: Restored Conda Environment to ${backupCondaEnvironment}: ${CONDA_DEFAULT_ENV}"
 fi

scripts/reports/ArtifactDependenciesJupyter.sh

@@ -36,4 +36,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the Jupyter Notebook "ArtifactDependencies.ipynb" within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/ArtifactDependencies.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/ArtifactDependencies.ipynb")

scripts/reports/ExternalDependenciesJupyter.sh

@@ -34,4 +34,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the following Jupyter Notebook within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/ExternalDependencies.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/ExternalDependencies.ipynb")

scripts/reports/InternalDependenciesJupyter.sh

@@ -33,4 +33,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the Jupyter Notebook "InternalDependencies.ipynb" within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/InternalDependencies.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/InternalDependencies.ipynb")

scripts/reports/MethodMetricsJupyter.sh

@@ -33,4 +33,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the following Jupyter Notebook within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/MethodMetrics.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/MethodMetrics.ipynb")

scripts/reports/NodeEmbeddingsJupyter.sh

@@ -34,4 +34,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the Jupyter Notebook "InternalDependencies.ipynb" within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/NodeEmbeddings.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/NodeEmbeddings.ipynb")

scripts/reports/ObjectOrientedDesignMetricsJupyter.sh

@@ -33,4 +33,4 @@ FULL_REPORT_DIRECTORY="${REPORTS_DIRECTORY}/${REPORT_NAME}"
 mkdir -p "${FULL_REPORT_DIRECTORY}"
 
 # Execute and convert the Jupyter Notebook "ObjectOrientedDesignMetrics.ipynb" within the given reports directory
-(cd "${FULL_REPORT_DIRECTORY}" && exec ${SCRIPTS_DIR}/executeJupyterNotebook.sh ${JUPYTER_NOTEBOOK_DIRECTORY}/ObjectOrientedDesignMetrics.ipynb)
+(cd "${FULL_REPORT_DIRECTORY}" && exec "${SCRIPTS_DIR}/executeJupyterNotebook.sh" "${JUPYTER_NOTEBOOK_DIRECTORY}/ObjectOrientedDesignMetrics.ipynb")