Merge pull request #57 from JohT/feature/windows-support

Support windows tested in git bash

Author: JohT

.github/workflows/code-structure-analysis.yml

@@ -97,6 +97,10 @@ jobs:
         auto-activate-base: false
         use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
 
+    - name: Conda environment info
+      shell: bash -el {0}
+      run: conda info
+
     - name: Setup temp directory if missing
       run: mkdir -p ./temp
 

COMMANDS.md

@@ -2,47 +2,11 @@
 
 ## Start an analysis
 
-1. Create a directory for all analysis projects
-
-    ```shell
-    mkdir temp
-    cd temp
-    ```
-
-1. Create a working directory for your specific analysis
-  
-    ```shell
-    mkdir MyFirstAnalysis
-    cd MyFirstAnalysis
-    ```
-
-1. Choose an initial password for Neo4j
-
-    ```shell
-    export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
-    ```
-
-1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards)
-
-    ```shell
-    mkdir artifacts
-    ```
-
-1. Move the artifacts you want to analyze into the `artifacts` directory
-
-1. Optionally run a predefined script to download artifacts
-
-    ```shell
-    ./../../scripts/downloader/downloadAxonFramework.sh <version>
-    ```
-
-1. Optionally use a script to download artifacts from Maven ([details](#download-maven-artifacts-to-analyze))
-
-1. Start the analysis
-
-    ```shell
-    ./../../scripts/analysis/analyze.sh
-    ```
+An analysis is started with the script [analyze.sh](./scripts/analysis/analyze.sh).
+To run all analysis steps simple execute the following command:
+```shell
+./../../scripts/analysis/analyze.sh
+```
 
 👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
 👉 See [Code Structure Analysis Pipeline](./.github/workflows/code-structure-analysis.yml) on how to do this within a GitHub Actions Workflow.

GETTING_STARTED.md

@@ -0,0 +1,73 @@
+# Code Graph Analysis Pipeline - Getting started guide
+
+This document describes the steps to get started as quickly as possible. 
+For more details on what you can do with this pipeline see [README](./README.md).
+For more details on how the commands work in detail see [COMMANDS](./COMMANDS.md).
+
+## 🛠 Prerequisites
+
+Please read through the [Prerequisites](./README.md#🛠-prerequisites) in the [README](./README.md) file for what is required to run the scripts.
+
+## Start an analysis
+
+1. Create a directory for all analysis projects
+
+    ```shell
+    mkdir temp
+    cd temp
+    ```
+
+1. Create a working directory for your specific analysis
+  
+    ```shell
+    mkdir MyFirstAnalysis
+    cd MyFirstAnalysis
+    ```
+
+1. Choose an initial password for Neo4j
+
+    ```shell
+    export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
+    ```
+
+1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards)
+
+    ```shell
+    mkdir artifacts
+    ```
+
+1. Move the artifacts you want to analyze into the `artifacts` directory
+
+1. Optionally run a predefined script to download artifacts
+
+    ```shell
+    ./../../scripts/downloader/downloadAxonFramework.sh <version>
+    ```
+
+1. Optionally use a script to download artifacts from Maven ([details](#download-maven-artifacts-to-analyze))
+
+1. Start the analysis
+
+  - Without any additional dependencies:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Csv
+    ```
+  - Jupyter notebook reports when Python and Conda are installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Jupyter
+    ```
+  - Graph visualizations when Node.js and npm are installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh --report Jupyter
+    ```
+  - All reports with Python, Conda, Node.js and npm installed:
+    ```shell
+    ./../../scripts/analysis/analyze.sh
+    ```
+  - To explore the database yourself without any automatically generated reports and no additional requirements:
+    ```shell
+    ./../../scripts/analysis/analyze.sh
+    ```
+
+👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
+👉 See [Code Structure Analysis Pipeline](./.github/workflows/code-structure-analysis.yml) on how to do this within a GitHub Actions Workflow.

README.md

@@ -6,7 +6,7 @@ Contained within this repository is a comprehensive and automated code graph ana
 
 ---
 
-## 🚀 Features
+## ✨ Features
 
 - Analyze static code structure as a graph
 - Fully automated [pipeline](./.github/workflows/code-structure-analysis.yml) from tool installation to report generation
@@ -37,13 +37,36 @@ Here are some reports that utilize Neo4j's [Graph Data Science Library](https://
 
 ## 🛠 Prerequisites
 
-- Java 17 is required (June 2023 Neo4j 5.x requirement)
-- Python and a conda package manager are required for Jupyter Notebook reports
-- Chromium will automatically be downloaded if needed for Jupyter Notebook reports in PDF format
+- Java 17 is [required for Neo4j](https://neo4j.com/docs/operations-manual/current/installation/requirements/#deployment-requirements-software) (Neo4j 5.x requirement).
+- On Windows it is recommended to use the git bash provided by [git for windows](https://gitforwindows.org).
+- [jq](https://github.com/jqlang/jq) the "lightweight and flexible command-line JSON processor" needs to be installed. Latest releases: https://github.com/jqlang/jq/releases/latest. Check using `jq --version`.
+- Set environment variable `NEO4J_INITIAL_PASSWORD` to a password of your choice. For example:
+  ```shell
+  export NEO4J_INITIAL_PASSWORD=neo4j_password_of_my_choice
+  ```
+  To run Jupyter notebooks, create an `.env` file in the folder from where you open the notebook containing for example: `NEO4J_INITIAL_PASSWORD=neo4j_password_of_my_choice`
+
+### Further Prerequisites for Python and Jupyter Notebooks
+
+- Python is required for Jupyter Notebook reports.
+- A conda package manager like [Miniconda](https://docs.conda.io/projects/miniconda/en/latest) or [Anaconda](https://www.anaconda.com/download)(Recommended for Windows) is required for Jupyter Notebook reports.
+- Chromium will automatically be downloaded if needed for Jupyter Notebook PDF reports generation.
+
+### Further Prerequisites for Graph Visualization
+
+These tools are needed to run the graph visualization scripts of directory [graph-visualization](./graph-visualization):
+
+- [Node.js](https://nodejs.org/en)
+- [npm](https://www.npmjs.com)
+
+### Hints for Windows
+
+- Add this line to your `~/.bashrc` file if you are using Anaconda3: `/c/ProgramData/Anaconda3/etc/profile.d/conda.sh`. Try to find a similar script for other conda package managers or versions.
+- Run `conda init` in the git bash opened as administrator. Running it in normal mode usually leads to an error message.
 
-## Getting Started
+## 🚀 Getting Started
 
-See [Start an analysis](./COMMANDS.md#start-an-analysis) in the [Commands Reference](./COMMANDS.md) on how to start an analysis on your local machine.
+See [GETTING_STARTED.md](./GETTING_STARTED.md) on how to get started on your local machine.
 
 ## 🏗 Pipeline and Tools
 

cypher/Community_Detection/Community_Detection_3a_WeaklyConnectedComponents_Estimate.cypher

@@ -23,4 +23,4 @@ RETURN requiredMemory
       ,heapPercentageMin
       ,heapPercentageMax
       ,treeView
-      ,mapView
+      //,mapView //doesn't work on Windows with git bash jq version jq-1.7-dirty

cypher/Community_Detection/Community_Detection_4a_Label_Propagation_Estimate.cypher

@@ -23,4 +23,4 @@ RETURN requiredMemory
       ,heapPercentageMin
       ,heapPercentageMax
       ,treeView
-      ,mapView
+      //,mapView //doesn't work on Windows with git bash jq version jq-1.7-dirty

cypher/Community_Detection/Community_Detection_5a_K_Core_Decomposition_Estimate.cypher

@@ -21,4 +21,4 @@ RETURN requiredMemory
       ,heapPercentageMin
       ,heapPercentageMax
       ,treeView
-      ,mapView
+      //,mapView //doesn't work on Windows with git bash jq version jq-1.7-dirty

cypher/Community_Detection/Community_Detection_6a_Approximate_Maximum_k_cut_Estimate.cypher

@@ -23,4 +23,4 @@ RETURN requiredMemory
       ,heapPercentageMin
       ,heapPercentageMax
       ,treeView
-      ,mapView
+      //,mapView //doesn't work on Windows with git bash jq version jq-1.7-dirty

cypher/External_Dependencies/External_package_usage_per_artifact_sorted_top.cypher

@@ -46,7 +46,6 @@ RETURN artifactName
       ,externalTypeRate
       ,numberOfExternalTypeCaller
       ,numberOfExternalTypeCalls
-      ,size(externalPackageNames) AS numberOfExternalPackages
-      ,externalPackageNames[0..4] AS top5ExternalPackages
-      ,externalTypeNames[0..1]    AS someExternalTypes
-LIMIT 40
+      ,size(externalPackageNames)                 AS numberOfExternalPackages
+      ,externalPackageNames[0..4]                 AS top5ExternalPackages
+      ,apoc.coll.flatten(externalTypeNames)[0..9] AS someExternalTypes

cypher/Node_Embeddings/Node_Embeddings_1a_Fast_Random_Projection_Estimate.cypher

@@ -6,5 +6,5 @@ CALL gds.fastRP.stream.estimate(
      ,relationshipWeightProperty: $dependencies_projection_weight_property
   }
 )
- YIELD requiredMemory, nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView, mapView
-RETURN requiredMemory, nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView, mapView
+ YIELD requiredMemory, nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView
+RETURN requiredMemory, nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView