Merge pull request #147 from JohT/fix/assure-that-tsne-perplexity-is-lower-than-the-sample-size-for-small-graphs

Assure that t-SNE perplexity parameter is lower than the sample size for small graphs

Author: JohT

COMMANDS.md

@@ -22,7 +22,7 @@
     - [Setup Neo4j Graph Database](#setup-neo4j-graph-database)
     - [Start Neo4j Graph Database](#start-neo4j-graph-database)
     - [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer)
-    - [Download Maven Artifacts to Analyze](#download-maven-artifacts-to-analyze)
+    - [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze)
     - [Reset the database and scan the java artifacts](#reset-the-database-and-scan-the-java-artifacts)
 - [Database Queries](#database-queries)
     - [Cypher Shell](#cypher-shell)
@@ -52,7 +52,8 @@ To run all analysis steps simple execute the following command:
 ./../../scripts/analysis/analyze.sh
 ```
 
-👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
+👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps for a Java Project.
+👉 See [scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) as an example script that combines all the above steps for a Typescript Project.  
 👉 See [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.yml) on how to do this within a GitHub Actions Workflow.
 
 ### Command Line Options

GETTING_STARTED.md

@@ -80,5 +80,6 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
 
     Then open your browser and login to your [local Neo4j Web UI](http://localhost:7474/browser) with "neo4j" as user and the initial password you've chosen.
 
-👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps.  
+👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps for a Java Project.
+👉 See [scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) as an example script that combines all the above steps for a Typescript Project. 
 👉 See [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.yml) on how to do this within a GitHub Actions Workflow.

README.md

@@ -26,13 +26,13 @@ Contained within this repository is a comprehensive and automated code graph ana
 
 Here is an overview of reports made with [Jupyter Notebooks](https://jupyter.org). For a detailed reference see [Jupyter Notebook Report Reference](#page_with_curl-jupyter-notebook-report-reference
 
-- [External Dependencies](./results/AxonFramework-4.9.3/external-dependencies/ExternalDependencies.md) contains detailed information about external library usage ([Notebook](./jupyter/ExternalDependenciesJava.ipynb)).
-- [Internal Dependencies](./results/AxonFramework-4.9.3/internal-dependencies/InternalDependencies.md) is based on [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html) and also includes cyclic dependencies ([Notebook](./jupyter/InternalDependenciesJava.ipynb)).
-- [Method Metrics](./results/AxonFramework-4.9.3/method-metrics/MethodMetrics.ipynb)  shows how the effective number of lines of code and the cyclomatic complexity are distributed across the methods in the code ([Notebook](./jupyter/MethodMetricsJava.ipynb)).
-- [Node Embeddings](./results/AxonFramework-4.9.3/node-embeddings/NodeEmbeddings.md) shows how to generate node embeddings and to further reduce their dimensionality to be able to visualize them in a 2D plot ([Notebook](./jupyter/NodeEmbeddingsJava.ipynb)).
-- [Object Oriented Design Quality Metrics](./results/AxonFramework-4.9.3/object-oriented-design-metrics/ObjectOrientedDesignMetrics.md) is based on [OO Design Quality Metrics by Robert Martin](https://api.semanticscholar.org/CorpusID:18246616) ([Notebook](./jupyter/ObjectOrientedDesignMetricsJava.ipynb)).
-- [Overview](./results/AxonFramework-4.9.3/overview/Overview.md) contains overall statistics and details about methods and their complexity. ([Notebook](./jupyter/OverviewJava.ipynb)).
-- [Visibility Metrics](./results/AxonFramework-4.9.3/visibility-metrics/VisibilityMetrics.md) ([Notebook](./jupyter/VisibilityMetricsJava.ipynb)).
+- [External Dependencies](./results/AxonFramework-4.9.3/external-dependencies-java/ExternalDependenciesJava.md) contains detailed information about external library usage ([Notebook](./jupyter/ExternalDependenciesJava.ipynb)).
+- [Internal Dependencies](./results/AxonFramework-4.9.3/internal-dependencies-java/InternalDependenciesJava.md) is based on [Analyze java package metrics in a graph database](https://joht.github.io/johtizen/data/2023/04/21/java-package-metrics-analysis.html) and also includes cyclic dependencies ([Notebook](./jupyter/InternalDependenciesJava.ipynb)).
+- [Method Metrics](./results/AxonFramework-4.9.3/method-metrics-java/MethodMetricsJava.md)  shows how the effective number of lines of code and the cyclomatic complexity are distributed across the methods in the code ([Notebook](./jupyter/MethodMetricsJava.ipynb)).
+- [Node Embeddings](./results/AxonFramework-4.9.3/node-embeddings-java/NodeEmbeddingsJava.md) shows how to generate node embeddings and to further reduce their dimensionality to be able to visualize them in a 2D plot ([Notebook](./jupyter/NodeEmbeddingsJava.ipynb)).
+- [Object Oriented Design Quality Metrics](./results/AxonFramework-4.9.3/object-oriented-design-metrics-java/ObjectOrientedDesignMetricsJava.md) is based on [OO Design Quality Metrics by Robert Martin](https://api.semanticscholar.org/CorpusID:18246616) ([Notebook](./jupyter/ObjectOrientedDesignMetricsJava.ipynb)).
+- [Overview](./results/AxonFramework-4.9.3/overview-java/OverviewJava.md) contains overall statistics and details about methods and their complexity. ([Notebook](./jupyter/OverviewJava.ipynb)).
+- [Visibility Metrics](./results/AxonFramework-4.9.3/visibility-metrics-java/VisibilityMetricsJava.md) ([Notebook](./jupyter/VisibilityMetricsJava.ipynb)).
 - [Wordcloud](./results/AxonFramework-4.9.3/wordcloud/Wordcloud.md) contains a visual representation of package and class names ([Notebook](./jupyter/Wordcloud.ipynb)).
 
 ### :book: Graph Data Science Reports

jupyter/NodeEmbeddingsJava.ipynb

@@ -258,9 +258,16 @@
     "    # See https://bobbyhadz.com/blog/python-attributeerror-list-object-has-no-attribute-shape\n",
     "    embeddings_as_numpy_array = np.array(embeddings.embedding.to_list())\n",
     "\n",
+    "    # The parameter \"perplexity\" needs to be smaller than the sample size\n",
+    "    # See https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html\n",
+    "    number_of_nodes=embeddings.shape[0]\n",
+    "    perplexity = min(number_of_nodes - 1.0, 30.0)\n",
+    "    print(\"t-SNE: Sample size (Number of nodes)={size}\".format(size = number_of_nodes))\n",
+    "    print(\"t-SNE: perplexity={perplexity}\".format(perplexity=perplexity))\n",
+    "\n",
     "    # Use t-distributed stochastic neighbor embedding (t-SNE) to reduce the dimensionality \n",
     "    # of the previously calculated node embeddings to 2 dimensions for visualization\n",
-    "    t_distributed_stochastic_neighbor_embedding = TSNE(n_components=2, verbose=1, random_state=50)\n",
+    "    t_distributed_stochastic_neighbor_embedding = TSNE(n_components=2, perplexity=perplexity, verbose=1, random_state=50)\n",
     "    two_dimension_node_embeddings = t_distributed_stochastic_neighbor_embedding.fit_transform(embeddings_as_numpy_array)\n",
     "    display(two_dimension_node_embeddings.shape) # Display the shape of the t-SNE result\n",
     "\n",

jupyter/NodeEmbeddingsTypescript.ipynb

@@ -258,9 +258,16 @@
     "    # See https://bobbyhadz.com/blog/python-attributeerror-list-object-has-no-attribute-shape\n",
     "    embeddings_as_numpy_array = np.array(embeddings.embedding.to_list())\n",
     "\n",
+    "    # The parameter \"perplexity\" needs to be smaller than the sample size\n",
+    "    # See https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html\n",
+    "    number_of_nodes=embeddings.shape[0]\n",
+    "    perplexity = min(number_of_nodes - 1.0, 30.0)\n",
+    "    print(\"t-SNE: Sample size (Number of nodes)={size}\".format(size = number_of_nodes))\n",
+    "    print(\"t-SNE: perplexity={perplexity}\".format(perplexity=perplexity))\n",
+    "\n",
     "    # Use t-distributed stochastic neighbor embedding (t-SNE) to reduce the dimensionality \n",
     "    # of the previously calculated node embeddings to 2 dimensions for visualization\n",
-    "    t_distributed_stochastic_neighbor_embedding = TSNE(n_components=2, verbose=1, random_state=50)\n",
+    "    t_distributed_stochastic_neighbor_embedding = TSNE(n_components=2, perplexity=perplexity, verbose=1, random_state=50)\n",
     "    two_dimension_node_embeddings = t_distributed_stochastic_neighbor_embedding.fit_transform(embeddings_as_numpy_array)\n",
     "    display(two_dimension_node_embeddings.shape) # Display the shape of the t-SNE result\n",
     "\n",

scripts/examples/analyzeReactRouter.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# This is an example for the analysis of a the Typescript project "react-router".
+# It includes the creation of the temporary directory, the working directory, the artifacts download and the analysis itself.
+
+# Note: The first (and only) parameter is the version of "react-router" to analyze.
+# Note: This script is meant to be started in the root directory of this repository.
+
+# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands)
+set -o errexit -o pipefail
+
+# Read the first input argument containing the version of the artifacts
+if [ "$#" -ne 1 ]; then
+  echo "analyzerReactRouter Error: Usage: $0 <version>" >&2
+  exit 1
+fi
+projectVersion=$1
+
+# Check if environment variable is set
+if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
+    echo "analyzerReactRouter: Error: Requires environment variable NEO4J_INITIAL_PASSWORD to be set first. Use 'export NEO4J_INITIAL_PASSWORD=<your-own-password>'."
+    exit 1
+fi
+
+# Create the temporary directory for all analysis projects.
+mkdir -p ./temp
+cd ./temp
+
+# Create the working directory for this specific analysis.
+mkdir -p "./react-router-${projectVersion}"
+cd "./react-router-${projectVersion}" 
+
+# Create the artifacts directory that will contain the code to be analyzed.
+mkdir -p ./artifacts
+
+# Download AxonFramework artifacts (jar files) from Maven
+./../../scripts/downloader/downloadReactRouter.sh "${projectVersion}"
+
+# Start the analysis
+./../../scripts/analysis/analyze.sh