Provide script to import git log as csv

JohT · JohT · commit 59a35a7a7aa5 · 2024-06-11T08:32:21.000+02:00
diff --git a/COMMANDS.md b/COMMANDS.md
@@ -24,6 +24,8 @@
     - [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer)
     - [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze)
     - [Reset the database and scan the java artifacts](#reset-the-database-and-scan-the-java-artifacts)
+    - [Import git log](#import-git-log)
+    - [Import aggregated git log](#import-aggregated-git-log)
 - [Database Queries](#database-queries)
     - [Cypher Shell](#cypher-shell)
     - [HTTP API](#http-api)
@@ -214,6 +216,27 @@ enhance the data further with relationships between artifacts and packages.
 
 Be aware that this script deletes all previous relationships and nodes in the local Neo4j Graph database.
 
+### Import git log
+
+Use [importGitLog.sh](./scripts/importGitLog.sh) to import git log data into the Graph.
+It uses `git log` to extract commits, their authors and the changed filenames into an intermediate CSV file that is then imported into Neo4j with the following schema:
+
+```Cypher
+(Git:Log:Author)-[:AUTHORED]->(Git:Log:Commit)->[:CONTAINS]->(Git:Log:File)
+```
+
+The optional parameter `--repository directory-path-to-a-git-repository` can be used to select a different directory for the repository. By default, the `source` directory within the analysis workspace directory is used. This command only needs the git history to be present so a `git clone --bare` is sufficient. If the `source` directory is also used for the analysis then a full git clone is of course needed (like for Typescript).
+
+👉**Note:** Commit messages containing `[bot]` are filtered out to ignore changes made by bots.
+
+### Import aggregated git log
+
+Use [importAggregatedGitLog.sh](./scripts/importAggregatedGitLog.sh) to import git log data in an aggregated form into the Graph. It works similar to the [full git log version above](#import-git-log). The only difference is that not every single commit is imported. Instead, changes are grouped per month including their commit count. This is in many cases sufficient and reduces data size and processing time significantly. Here is the resulting schema:
+
+```Cypher
+(Git:Log:Author)-[:AUTHORED]->(Git:Log:ChangeSpan)-[:CONTAINS]->(Git:Log:File)
+```
+
 ## Database Queries
 
 ### Cypher Shell
diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md
@@ -24,7 +24,7 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
     cd MyFirstAnalysis
     ```
 
-1. Choose an initial password for Neo4j
+1. Choose an initial password for Neo4j if not already done
 
     ```shell
     export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
@@ -36,9 +36,11 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
     mkdir artifacts
     ```
 
-1. Move the artifacts you want to analyze into the `artifacts` directory
+1. Move the artifacts (Java jar or Typescript analysis json files) you want to analyze into the `artifacts` directory
 
-1. Optionally run a predefined script to download artifacts
+1. Optionally, create a `source` directory and clone the corresponding source code into it to also gather git log data.
+
+1. Alternatively to the steps above, run an already predefined download script
 
     ```shell
     ./../../scripts/downloader/downloadAxonFramework.sh <version>
@@ -48,31 +50,31 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
 
 1. Start the analysis
 
-  - Without any additional dependencies:
+   - Without any additional dependencies:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Csv
     ```
 
-  - Jupyter notebook reports when Python and Conda are installed:
+   - Jupyter notebook reports when Python and Conda are installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Jupyter
     ```
 
-  - Graph visualizations when Node.js and npm are installed:
+   - Graph visualizations when Node.js and npm are installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Jupyter
     ```
 
-  - All reports with Python, Conda, Node.js and npm installed:
+   - All reports with Python, Conda, Node.js and npm installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh
     ```
 
-  - To explore the database yourself without any automatically generated reports and no additional requirements:
+   - To explore the database yourself without any automatically generated reports and no additional requirements:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --explore
diff --git a/README.md b/README.md
@@ -91,7 +91,9 @@ This could be as simple as running the following command in your Typescript proj
   npx --yes @jqassistant/ts-lce
   ```
 
-- Copy the resulting json file (e.g. `.reports/jqa/ts-output.json`) into the "artifacts" directory for your analysis work directory. Custom subdirectories within "artifacts" are also supported.
+- It is recommended to put the cloned source code repository into a directory called `source` within the analysis workspace so that it will also be picked up to import git log data.
+
+- Copy the resulting json file (e.g. `.reports/jqa/ts-output.json`) into the `artifacts` directory for your analysis work directory. Custom subdirectories within `artifacts` are also supported.
 
 ## :rocket: Getting Started
 
@@ -105,7 +107,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
 - [Checkout GIT Repository](https://github.com/actions/checkout)
 - [Setup Java](https://github.com/actions/setup-java)
 - [Setup Python with Conda](https://github.com/conda-incubator/setup-miniconda) package manager [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge)
-- Download artifacts that contain the code to be analyzed [scripts/artifacts](./scripts/downloader/)
+- Download artifacts and optionally source code that contain the code to be analyzed [scripts/downloader](./scripts/downloader)
 - Setup [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
 - Setup [jQAssistant](https://jqassistant.github.io/jqassistant/doc) for Java and [Typescript](https://github.com/jqassistant-plugin/jqassistant-typescript-plugin) analysis ([analysis.sh](./scripts/analysis/analyze.sh))
 - Start [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
@@ -176,7 +178,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
   👉 The script will automatically be included because of the directory and its name ending with "Jupyter.sh".
 
 - How can i add another code basis to be analyzed automatically?  
-  👉 Create a new artifacts download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
+  👉 Create a new download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
   👉 Run the script separately before executing [analyze.sh](./scripts/analysis/analyze.sh) also in the [pipeline](./.github/workflows/java-code-analysis.yml).
 
 - How can i trigger a full re-scan of all artifacts?  
diff --git a/cypher/GitLog/Add_RESOLVES_TO_relationships_to_git_files.cypher b/cypher/GitLog/Add_RESOLVES_TO_relationships_to_git_files.cypher
@@ -0,0 +1,12 @@
+// Connect git files to Java files with a RESOLVES_TO relationship if their names match
+
+MATCH (file:File&!Git)
+ WITH file
+     ,replace(file.fileName, '.class', '.java') AS fileName
+MATCH (git_file:File&Git)
+WHERE git_file.fileName ENDS WITH fileName
+MERGE (git_file)-[:RESOLVES_TO]->(file)
+  SET git_file.resolved = true
+RETURN labels(file)[0..4]                AS labels
+       ,count(DISTINCT fileName)         AS numberOfFileNames
+       ,collect(DISTINCT fileName)[0..4] AS examples
diff --git a/cypher/GitLog/Delete_git_log_data.cypher b/cypher/GitLog/Delete_git_log_data.cypher
@@ -0,0 +1,7 @@
+// Delete all Git log data in the Graph
+
+MATCH (n:Git)
+CALL { WITH n
+DETACH DELETE n
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(n) as numberOfDeletedRows
diff --git a/cypher/GitLog/Import_aggregated_git_log_csv_data.cypher b/cypher/GitLog/Import_aggregated_git_log_csv_data.cypher
@@ -0,0 +1,17 @@
+// Import aggregated git log CSV data with the following schema: (Git:Log:Author)-[:AUTHORED]->(Git:Log:ChangeSpan)-[:CONTAINS]->(Git:Log:File)
+
+LOAD CSV WITH HEADERS FROM "file:///aggregatedGitLog.csv" AS row
+CALL { WITH row
+    MERGE (git_author:Git:Log:Author {name: row.author, email: row.email})
+    MERGE (git_change_span:Git:Log:ChangeSpan {
+        year:    toInteger(row.year),
+        month:   toInteger(row.month),
+        commits: toInteger(row.commits)
+    })
+    MERGE (git_file:Git:Log:File {fileName: row.filename})
+    MERGE (git_author)-[:AUTHORED]->(git_change_span)
+    MERGE (git_change_span)-[:CONTAINS]->(git_file)
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(DISTINCT row.author)   AS numberOfAuthors
+      ,count(DISTINCT row.filename) AS numberOfFiles
+      ,sum(toInteger(row.commits))  AS numberOfCommits
diff --git a/cypher/GitLog/Import_git_log_csv_data.cypher b/cypher/GitLog/Import_git_log_csv_data.cypher
@@ -0,0 +1,18 @@
+// Import git log CSV data with the following schema: (Git:Log:Author)-[:AUTHORED]->(Git:Log:Commit)-[:CONTAINS]->(Git:Log:File)
+
+LOAD CSV WITH HEADERS FROM "file:///gitLog.csv" AS row
+CALL { WITH row
+    MERGE (git_author:Git:Log:Author {name: row.author, email: row.email})
+    MERGE (git_commit:Git:Log:Commit {
+        hash:           row.hash, 
+        message:        row.message,
+        timestamp:      datetime(row.timestamp),
+        timestamp_unix: toInteger(row.timestamp_unix)
+    })
+    MERGE (git_file:Git:Log:File {fileName: row.filename})
+    MERGE (git_author)-[:AUTHORED]->(git_commit)
+    MERGE (git_commit)-[:CONTAINS]->(git_file)
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(DISTINCT row.author)   AS numberOfAuthors
+      ,count(DISTINCT row.filename) AS numberOfFiles
+      ,count(DISTINCT row.hash)     AS numberOfCommits
diff --git a/cypher/GitLog/Index_author_name.cypher b/cypher/GitLog/Index_author_name.cypher
@@ -0,0 +1,3 @@
+// Create index for author name (git data)
+
+CREATE INDEX INDEX_AUTHOR_NAME IF NOT EXISTS FOR (n:Author) ON (n.name)
diff --git a/cypher/GitLog/Index_change_span_year.cypher b/cypher/GitLog/Index_change_span_year.cypher
@@ -0,0 +1,3 @@
+// Create index for change span year (aggregated git data)
+
+CREATE INDEX INDEX_CHANGE_SPAN_YEAR IF NOT EXISTS FOR (n:ChangeSpan) ON (n.year)
diff --git a/cypher/GitLog/Index_commit_hash.cypher b/cypher/GitLog/Index_commit_hash.cypher
@@ -0,0 +1,3 @@
+// Create index for commit hash (git data)
+
+CREATE INDEX INDEX_COMMIT_HASH IF NOT EXISTS FOR (n:Commit) ON (n.hash)
diff --git a/cypher/GitLog/Index_file_name.cypher b/cypher/GitLog/Index_file_name.cypher
@@ -0,0 +1,3 @@
+// Create index for the file name
+
+CREATE INDEX INDEX_FILE_NAME IF NOT EXISTS FOR (t:File) ON (t.fileName)
diff --git a/cypher/GitLog/Set_number_of_git_commits_for_Java_types.cypher b/cypher/GitLog/Set_number_of_git_commits_for_Java_types.cypher
@@ -0,0 +1,9 @@
+// Set numberOfGitCommits property on Java Types
+
+MATCH (java_type:Java:Type)<-[:RESOLVES_TO]-(git_file:Git:File)
+OPTIONAL MATCH (git_file)<-[:CONTAINS]-(git_commit:Git:Commit)
+OPTIONAL MATCH (git_file)<-[:CONTAINS]-(git_changespan:Git:ChangeSpan)
+ WITH java_type
+     ,coalesce(nullif(count(git_commit),0), nullif(sum(git_changespan.commits),0)) AS numberOfGitCommits
+  SET java_type.numberOfGitCommits = numberOfGitCommits
+RETURN count(DISTINCT java_type.fqn) AS typesChanges
diff --git a/scripts/importAggregatedGitLog.sh b/scripts/importAggregatedGitLog.sh
@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+
+# Uses git log to create a comma separated values (CSV) file containing aggregated changes, their author name and email address, year and month for all the files that were changed. The CSV is then imported into Neo4j.
+
+# Note: This script needs the path to a git repository directory. It defaults to SOURCE_DIRECTORY ("source"). 
+# Note: Import will be skipped without an error if the directory is not a git repository.
+# Note: This script needs git to be installed.
+
+# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands)
+set -o errexit -o pipefail
+
+# Overrideable Defaults
+NEO4J_EDITION=${NEO4J_EDITION:-"community"} # Choose "community" or "enterprise"
+NEO4J_VERSION=${NEO4J_VERSION:-"5.16.0"}
+TOOLS_DIRECTORY=${TOOLS_DIRECTORY:-"tools"} # Get the tools directory (defaults to "tools")
+SOURCE_DIRECTORY=${SOURCE_DIRECTORY:-"source"} # Get the source repository directory (defaults to "source")
+
+# Default and initial values for command line options
+repository="${SOURCE_DIRECTORY}"
+
+# Read  command line options
+USAGE="importAggregatedGitLog: Usage: $0 [--repository <git repository directory>(default=source)]"
+while [[ $# -gt 0 ]]; do
+  key="$1"
+  case $key in
+    --repository)
+      repository="$2"
+      # Check if the explicitly given repository is a valid directory
+      if [ ! -d "${repository}" ] ; then
+        echo "importAggregatedGitLog: Error: The given repository <${repository}> is not a directory" >&2
+        echo "${USAGE}" >&2
+        exit 1
+      fi
+      shift
+      ;;
+    *)
+      echo "importAggregatedGitLog: Error: Unknown option: ${key}"
+      echo "${USAGE}" >&2
+      exit 1
+  esac
+  shift
+done
+
+# Check if the repository is actually a git repository
+if ! (cd "${repository}" || exit; git rev-parse --git-dir 2> /dev/null || exit); then
+  echo "importAggregatedGitLog: Import skipped. ${repository} is not a git repository."
+  exit 0
+fi
+
+echo "importAggregatedGitLog: repository=${repository}"
+
+## Get this "scripts" directory if not already set
+# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution. 
+# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
+# This way non-standard tools like readlink aren't needed.
+SCRIPTS_DIR=${SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )} # Repository directory containing the shell scripts
+echo "importAggregatedGitLog: SCRIPTS_DIR=$SCRIPTS_DIR"
+
+# Get the "cypher" directory by taking the path of this script and going two directory up and then to "cypher".
+CYPHER_DIR=${CYPHER_DIR:-"${SCRIPTS_DIR}/../cypher"}
+echo "importAggregatedGitLog: CYPHER_DIR=${CYPHER_DIR}"
+
+# Define functions (like execute_cypher and execute_cypher_summarized) to execute cypher queries from within a given file
+source "${SCRIPTS_DIR}/executeQueryFunctions.sh"
+
+# Internal constants
+IMPORTS_CYPHER_DIR="${CYPHER_DIR}/Imports"
+NEO4J_INSTALLATION_NAME="neo4j-${NEO4J_EDITION}-${NEO4J_VERSION}"
+NEO4J_INSTALLATION_DIRECTORY="${TOOLS_DIRECTORY}/${NEO4J_INSTALLATION_NAME}"
+NEO4J_FULL_IMPORT_DIRECTORY=$(cd "${NEO4J_INSTALLATION_DIRECTORY}/import"; pwd)
+OUTPUT_CSV_FILENAME="${NEO4J_FULL_IMPORT_DIRECTORY}/aggregatedGitLog.csv"
+
+# ----- Create a CSV file with git log data containing all commits and their changed files
+echo "importAggregatedGitLog: Creating ${OUTPUT_CSV_FILENAME} from git log..."
+
+(
+  # Git log needs to be executed in the directory of the repository.
+  # This is done in a sub shell to automatically return to the previous directory. 
+  cd "${repository}" || exit
+
+  # Prints the header line of the CSV file with the names of the columns.
+  echo "filename,year,month,author,email,commits" > "${OUTPUT_CSV_FILENAME}"
+  
+  # Prints the aggregated git log in CSV format starting with the changed file, year-month, author, author email and number of commits.
+  # Includes quoted strings, double quote escaping and supports commas in strings.
+  git log --no-merges --pretty=format:' %ad,,,%an,,,%ae' --date=format:'%Y,%m' --name-only | \
+  awk 'BEGIN { COMMA=",";QUOTE="\"" } /^ / { split($0, a, ",,,"); gsub(/^ /, "", a[1]); gsub(/"/, "\"\"", a[2]); gsub(/"/, "\"\"", a[3]); commit=a[1] COMMA QUOTE a[2] QUOTE COMMA QUOTE a[3] QUOTE } NF && !/^\ / { print "\""$0"\"," commit }' |
+  grep -v -F '[bot]' | \
+  sort | uniq -c | \
+  sed -E 's/^ *([0-9]+) (.+)/\2,\1/g' \
+      >> "${OUTPUT_CSV_FILENAME}"
+  # Explanation:
+  #
+  # - --no-merges: Excludes merge commits from the log.
+  # - %ad: Author date (formatted as specified later)
+  # - %an: Author name
+  # - %ae: Author email
+  # - %ct: Commit date, Unix timestamp
+  # - %s: Subject of the commit
+  # - --date=format:'%Y,%m': Takes the year and the month of the date separated by a comma for example 2024,06
+  # - --name-only: Lists the files affected by each commit.
+  # - --pretty=format starts with a space that is needed to detect the start of a line.
+  # - The chosen delimiters ,,, are used to separate these fields to make parsing easier.
+  #   It is very unlikely that they appear in the contents and will be used as an intermediate step before escaping.
+  #
+  # - BEGIN { COMMA=","; QUOTE="\"" }: Initializes the variables COMMA and QUOTE to hold a comma and a double-quote character respectively.
+  # - /^ / { ... }: Processes lines that start with a space (indicating a file name in git log --name-only output).
+  # - gsub(/^ /, "", a[1]): Removes leading spaces from the first field (commit hash) that was used to indicate a new commit.
+  # - gsub(/"/, "\"\"", a[2]) escapes double quotes with two double quotes (CSV standard).
+  #   a[2] is the commit author. Double quote escaping is done for every string column
+  # - commit=...: Constructs the commit information in CSV format, including the year-month of the change, quoted author name, and email.
+  # - NF && !/^\ / { print "\""$0"\"," commit }: For non-empty lines that do not start with a space (indicating commit information), 
+  #   it prints the commit information followed by the file name(s), enclosed in quotes.
+  #
+  # - grep -v -F '[bot]': Filters out commits where the commit message includes [bot]
+  #   Used to identify commits made by automated systems or bots.
+  #
+  # - sort | uniq -c: Sorts the lines by their content (order of columns essential for that), removes duplicate lines and adds the number of duplicates at the beginning of each line
+  #-   sed -E 's/^ *([0-9]+) (.+)/\2,\1/g': Reformats each line so that the commits count are the last column delimited by a comma.
+)
+
+csv_file_size=$(wc -c "${OUTPUT_CSV_FILENAME}" | awk '{print $1}')
+csv_lines=$(wc -l "${OUTPUT_CSV_FILENAME}" | awk '{print $1}')
+echo "importAggregatedGitLog: File ${OUTPUT_CSV_FILENAME} with ${csv_file_size} bytes and ${csv_lines} lines created."
+# ---------
+
+# ----- Import aggregate git log data csv
+GIT_LOG_CYPHER_DIR="${CYPHER_DIR}/GitLog"
+
+echo "importAggregatedGitLog: Prepare import by creating indexes..."
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_author_name.cypher"
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_change_span_year.cypher"
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_file_name.cypher"
+
+echo "importAggregatedGitLog: Deleting all existing git data in the Graph..."
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Delete_git_log_data.cypher"
+
+echo "importAggregatedGitLog: Importing aggregated git log data into the Graph..."
+time execute_cypher "${GIT_LOG_CYPHER_DIR}/Import_aggregated_git_log_csv_data.cypher"
+
+echo "importAggregatedGitLog: Creating connections to nodes with matching file names..."
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Add_RESOLVES_TO_relationships_to_git_files.cypher"
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Add_RESOLVES_TO_relationships_to_git_files.cypher"
+execute_cypher "${GIT_LOG_CYPHER_DIR}/Set_number_of_git_commits_for_Java_types.cypher"
+# ---------
diff --git a/scripts/importGitLog.sh b/scripts/importGitLog.sh

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+// Create index for author name (git data)`
	`2`	`+`
	`3`	`+CREATE INDEX INDEX_AUTHOR_NAME IF NOT EXISTS FOR (n:Author) ON (n.name)`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+// Create index for change span year (aggregated git data)`
	`2`	`+`
	`3`	`+CREATE INDEX INDEX_CHANGE_SPAN_YEAR IF NOT EXISTS FOR (n:ChangeSpan) ON (n.year)`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+// Create index for commit hash (git data)`
	`2`	`+`
	`3`	`+CREATE INDEX INDEX_COMMIT_HASH IF NOT EXISTS FOR (n:Commit) ON (n.hash)`