Provide script to import git log as csv

Author: JohT

COMMANDS.md

@@ -24,6 +24,10 @@
     - [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer)
     - [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze)
     - [Reset the database and scan the java artifacts](#reset-the-database-and-scan-the-java-artifacts)
+    - [Import git log](#import-git-log)
+        - [Parameters](#parameters)
+        - [Resolving git files to code files](#resolving-git-files-to-code-files)
+    - [Import aggregated git log](#import-aggregated-git-log)
 - [Database Queries](#database-queries)
     - [Cypher Shell](#cypher-shell)
     - [HTTP API](#http-api)
@@ -214,6 +218,35 @@ enhance the data further with relationships between artifacts and packages.
 
 Be aware that this script deletes all previous relationships and nodes in the local Neo4j Graph database.
 
+### Import git log
+
+Use [importGitLog.sh](./scripts/importGitLog.sh) to import git log data into the Graph.
+It uses `git log` to extract commits, their authors and the names of the files changed with them. These are stored in an intermediate CSV file and are then imported into Neo4j with the following schema:
+
+```Cypher
+(Git:Log:Author)-[:AUTHORED]->(Git:Log:Commit)->[:CONTAINS]->(Git:Log:File)
+```
+
+👉**Note:** Commit messages containing `[bot]` are filtered out to ignore changes made by bots.
+
+#### Parameters
+
+The optional parameter `--repository directory-path-to-a-git-repository` can be used to select a different directory for the repository. By default, the `source` directory within the analysis workspace directory is used. This command only needs the git history to be present so a `git clone --bare` is sufficient. If the `source` directory is also used for the analysis then a full git clone is of course needed (like for Typescript).
+
+#### Resolving git files to code files
+
+After git log data has been imported successfully, [Add_RESOLVES_TO_relationships_to_git_files.cypher](./cypher/GitLog/Add_RESOLVES_TO_relationships_to_git_files.cypher) is used to try to resolve the imported git file names to  code files. This first attempt will cover most cases, but not all of them. With this approach it is, for example, not possible to distinguish identical file names in different Java jars from the git source files of a mono repo.
+
+You can use [List_unresolved_git_files.cypher](./cypher/GitLog/List_unresolved_git_files.cypher) to find code files that couldn't be matched to git file names and [List_ambiguous_git_files.cypher](./cypher/GitLog/List_ambiguous_git_files.cypher) to find ambiguously resolved git files. If you have any idea on how to improve this feel free to [open an issue](https://github.com/JohT/code-graph-analysis-pipeline/issues/new).
+
+### Import aggregated git log
+
+Use [importAggregatedGitLog.sh](./scripts/importAggregatedGitLog.sh) to import git log data in an aggregated form into the Graph. It works similar to the [full git log version above](#import-git-log). The only difference is that not every single commit is imported. Instead, changes are grouped per month including their commit count. This is in many cases sufficient and reduces data size and processing time significantly. Here is the resulting schema:
+
+```Cypher
+(Git:Log:Author)-[:AUTHORED]->(Git:Log:ChangeSpan)-[:CONTAINS]->(Git:Log:File)
+```
+
 ## Database Queries
 
 ### Cypher Shell

GETTING_STARTED.md

@@ -24,7 +24,7 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
     cd MyFirstAnalysis
     ```
 
-1. Choose an initial password for Neo4j
+1. Choose an initial password for Neo4j if not already done
 
     ```shell
     export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
@@ -36,9 +36,11 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
     mkdir artifacts
     ```
 
-1. Move the artifacts you want to analyze into the `artifacts` directory
+1. Move the artifacts (Java jar or Typescript analysis json files) you want to analyze into the `artifacts` directory
 
-1. Optionally run a predefined script to download artifacts
+1. Optionally, create a `source` directory and clone the corresponding source code into it to also gather git log data.
+
+1. Alternatively to the steps above, run an already predefined download script
 
     ```shell
     ./../../scripts/downloader/downloadAxonFramework.sh <version>
@@ -48,31 +50,31 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
 
 1. Start the analysis
 
-  - Without any additional dependencies:
+   - Without any additional dependencies:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Csv
     ```
 
-  - Jupyter notebook reports when Python and Conda are installed:
+   - Jupyter notebook reports when Python and Conda are installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Jupyter
     ```
 
-  - Graph visualizations when Node.js and npm are installed:
+   - Graph visualizations when Node.js and npm are installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --report Jupyter
     ```
 
-  - All reports with Python, Conda, Node.js and npm installed:
+   - All reports with Python, Conda, Node.js and npm installed:
 
     ```shell
     ./../../scripts/analysis/analyze.sh
     ```
 
-  - To explore the database yourself without any automatically generated reports and no additional requirements:
+   - To explore the database yourself without any automatically generated reports and no additional requirements:
 
     ```shell
     ./../../scripts/analysis/analyze.sh --explore

README.md

@@ -91,7 +91,9 @@ This could be as simple as running the following command in your Typescript proj
   npx --yes @jqassistant/ts-lce
   ```
 
-- Copy the resulting json file (e.g. `.reports/jqa/ts-output.json`) into the "artifacts" directory for your analysis work directory. Custom subdirectories within "artifacts" are also supported.
+- It is recommended to put the cloned source code repository into a directory called `source` within the analysis workspace so that it will also be picked up to import git log data.
+
+- Copy the resulting json file (e.g. `.reports/jqa/ts-output.json`) into the `artifacts` directory for your analysis work directory. Custom subdirectories within `artifacts` are also supported.
 
 ## :rocket: Getting Started
 
@@ -105,7 +107,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
 - [Checkout GIT Repository](https://github.com/actions/checkout)
 - [Setup Java](https://github.com/actions/setup-java)
 - [Setup Python with Conda](https://github.com/conda-incubator/setup-miniconda) package manager [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge)
-- Download artifacts that contain the code to be analyzed [scripts/artifacts](./scripts/downloader/)
+- Download artifacts and optionally source code that contain the code to be analyzed [scripts/downloader](./scripts/downloader)
 - Setup [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
 - Setup [jQAssistant](https://jqassistant.github.io/jqassistant/doc) for Java and [Typescript](https://github.com/jqassistant-plugin/jqassistant-typescript-plugin) analysis ([analysis.sh](./scripts/analysis/analyze.sh))
 - Start [Neo4j](https://neo4j.com) Graph Database ([analysis.sh](./scripts/analysis/analyze.sh))
@@ -176,7 +178,7 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
   👉 The script will automatically be included because of the directory and its name ending with "Jupyter.sh".
 
 - How can i add another code basis to be analyzed automatically?  
-  👉 Create a new artifacts download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
+  👉 Create a new download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.  
   👉 Run the script separately before executing [analyze.sh](./scripts/analysis/analyze.sh) also in the [pipeline](./.github/workflows/java-code-analysis.yml).
 
 - How can i trigger a full re-scan of all artifacts?  

cypher/GitLog/Add_RESOLVES_TO_relationships_to_git_files_for_Java.cypher

@@ -0,0 +1,11 @@
+// Connect git files to Java files with a RESOLVES_TO relationship if their names match
+
+MATCH (file:File&!Git)
+ WITH file, replace(file.fileName, '.class', '.java') AS fileName
+MATCH (git_file:File&Git)
+WHERE git_file.fileName ENDS WITH fileName
+MERGE (git_file)-[:RESOLVES_TO]->(file)
+  SET git_file.resolved = true
+RETURN labels(file)[0..4]               AS labels
+      ,count(DISTINCT fileName)         AS numberOfFileNames
+      ,collect(DISTINCT fileName)[0..4] AS examples

cypher/GitLog/Delete_git_log_data.cypher

@@ -0,0 +1,7 @@
+// Delete all Git log data in the Graph
+
+MATCH (n:Git)
+CALL { WITH n
+DETACH DELETE n
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(n) as numberOfDeletedRows

cypher/GitLog/Import_aggregated_git_log_csv_data.cypher

@@ -0,0 +1,17 @@
+// Import aggregated git log CSV data with the following schema: (Git:Log:Author)-[:AUTHORED]->(Git:Log:ChangeSpan)-[:CONTAINS]->(Git:Log:File)
+
+LOAD CSV WITH HEADERS FROM "file:///aggregatedGitLog.csv" AS row
+CALL { WITH row
+    MERGE (git_author:Git:Log:Author {name: row.author, email: row.email})
+    MERGE (git_change_span:Git:Log:ChangeSpan {
+        year:    toInteger(row.year),
+        month:   toInteger(row.month),
+        commits: toInteger(row.commits)
+    })
+    MERGE (git_file:Git:Log:File {fileName: row.filename})
+    MERGE (git_author)-[:AUTHORED]->(git_change_span)
+    MERGE (git_change_span)-[:CONTAINS]->(git_file)
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(DISTINCT row.author)   AS numberOfAuthors
+      ,count(DISTINCT row.filename) AS numberOfFiles
+      ,sum(toInteger(row.commits))  AS numberOfCommits

cypher/GitLog/Import_git_log_csv_data.cypher

@@ -0,0 +1,18 @@
+// Import git log CSV data with the following schema: (Git:Log:Author)-[:AUTHORED]->(Git:Log:Commit)-[:CONTAINS]->(Git:Log:File)
+
+LOAD CSV WITH HEADERS FROM "file:///gitLog.csv" AS row
+CALL { WITH row
+    MERGE (git_author:Git:Log:Author {name: row.author, email: row.email})
+    MERGE (git_commit:Git:Log:Commit {
+        hash:           row.hash, 
+        message:        row.message,
+        timestamp:      datetime(row.timestamp),
+        timestamp_unix: toInteger(row.timestamp_unix)
+    })
+    MERGE (git_file:Git:Log:File {fileName: row.filename})
+    MERGE (git_author)-[:AUTHORED]->(git_commit)
+    MERGE (git_commit)-[:CONTAINS]->(git_file)
+} IN TRANSACTIONS OF 1000 ROWS
+RETURN count(DISTINCT row.author)   AS numberOfAuthors
+      ,count(DISTINCT row.filename) AS numberOfFiles
+      ,count(DISTINCT row.hash)     AS numberOfCommits

cypher/GitLog/Index_author_name.cypher

@@ -0,0 +1,3 @@
+// Create index for author name (git data)
+
+CREATE INDEX INDEX_AUTHOR_NAME IF NOT EXISTS FOR (n:Author) ON (n.name)

cypher/GitLog/Index_change_span_year.cypher

@@ -0,0 +1,3 @@
+// Create index for change span year (aggregated git data)
+
+CREATE INDEX INDEX_CHANGE_SPAN_YEAR IF NOT EXISTS FOR (n:ChangeSpan) ON (n.year)

cypher/GitLog/Index_commit_hash.cypher

@@ -0,0 +1,3 @@
+// Create index for commit hash (git data)
+
+CREATE INDEX INDEX_COMMIT_HASH IF NOT EXISTS FOR (n:Commit) ON (n.hash)