Skip to content

Commit 809eedd

Browse files
authored
Merge pull request #265 from JohT/feature/improve-typescript-scanning-and-git-import
Improve Typescript scanning and git import
2 parents 7d1ed6b + 8f3ba20 commit 809eedd

File tree

7 files changed

+119
-60
lines changed

7 files changed

+119
-60
lines changed

GETTING_STARTED.md

Lines changed: 43 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Code Graph Analysis Pipeline - Getting started guide
22

3-
This document describes the steps to get started as quickly as possible.
4-
For more details on what you can do with this pipeline see [README](./README.md).
5-
For more details on how the commands work in detail see [COMMANDS](./COMMANDS.md).
3+
This document describes the steps to get started as quickly as possible.
4+
👉 For more details on what else you can do see [README](./README.md).
5+
👉 For more details on how the commands work in detail see [COMMANDS](./COMMANDS.md).
66

77
## 🛠 Prerequisites
88

@@ -12,11 +12,11 @@ Please read through the [Prerequisites](./README.md#hammer_and_wrench-prerequisi
1212

1313
Just run one of the following examples in the directory of this file:
1414

15-
- [./scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) (Java event-sourcing library)
16-
- [./scripts/examples/analyzeAntDesign.sh](./scripts/examples/analyzeAntDesign.sh) (Typescript UI library)
17-
- [./scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) (Typescript React library)
15+
- Java Event-Sourcing Framework: [./scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh)
16+
- Typescript UI Library: [./scripts/examples/analyzeAntDesign.sh](./scripts/examples/analyzeAntDesign.sh)
17+
- Typescript React Library: [./scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh)
1818

19-
Use these optional command line options as you like:
19+
Use these optional command line options as needed:
2020

2121
- (Recommended) Only create CSV reports and skip Python and Node.js dependent reports. Example:
2222

@@ -38,81 +38,72 @@ Use these optional command line options as you like:
3838

3939
## Start an own analysis
4040

41-
1. Create a directory for all analysis projects.
41+
### 1. Setup
4242

43-
```shell
44-
mkdir temp
45-
cd temp
46-
```
43+
- Have a look at the [prerequisites](./README.md#hammer_and_wrench-prerequisites).
4744

48-
1. Create a working directory for your specific analysis.
49-
50-
```shell
51-
mkdir MyFirstAnalysis
52-
cd MyFirstAnalysis
53-
```
54-
55-
1. Choose an initial password for Neo4j if not already done.
45+
- Choose an initial password for Neo4j if not already done.
5646

5747
```shell
5848
export NEO4J_INITIAL_PASSWORD=theinitialpasswordthatihavechosenforneo4j
5949
```
6050

61-
1. Create the `artifacts` directory for the code to be analyzed (without `cd` afterwards).
51+
- Initialize you analysis project using [./init.sh](./init.sh).
6252

6353
```shell
64-
mkdir artifacts
54+
./init.sh MyAnalysisProjectName
6555
```
6656

67-
1. Move the artifacts (e.g. Java jars json files) you want to analyze into the `artifacts` directory.
57+
### 2. Prepare the code to be analyzed
58+
59+
- Move the artifacts (e.g. Java jars json files) you want to analyze into the `artifacts` directory.
6860

69-
1. If you want to analyze Typescript code, create a symbolic link inside the `source` directory that points to the Typescript project or copy the project into it.
61+
- If you want to analyze Typescript code, create a symbolic link inside the `source` directory that points to the Typescript project. Alternatively you can also copy the project into the `source` directory.
7062

71-
1. If you want to include git data like changed files and authors, create a symbolic link inside the `source` directory that points to the repository or clone it in the `source` directory. If you already have your Typescript project in there, you of course don't have to do it twice. If you are analyzing Java artifacts (no source needed), it is sufficient to use a bare clone that only contains the git history without the sources using `git clone --bare`.
63+
- If you want to include git data like changed files and authors, create a symbolic link inside the `source` directory that points to the repository or clone it into the `source` directory. If you already have your Typescript project in there, you of course don't have to do it twice. If you are analyzing Java artifacts (full source not needed), it is sufficient to use a bare clone that only contains the git history without the sources using `git clone --bare`.
7264
73-
1. Alternatively to the steps above, run an already predefined download script
65+
- Alternatively to the steps above, run an already predefined download script
7466
7567
```shell
7668
./../../scripts/downloader/downloadAxonFramework.sh <version>
7769
```
7870
79-
1. Optionally use a script to download artifacts from Maven ([details](./COMMANDS.md#download-maven-artifacts-to-analyze)).
71+
- Optionally use a script to download artifacts from Maven ([details](./COMMANDS.md#download-maven-artifacts-to-analyze)).
8072
81-
1. Start the analysis.
73+
### Start the analysis
8274
83-
- Without any additional dependencies:
75+
- Without any additional dependencies:
8476
85-
```shell
86-
./../../scripts/analysis/analyze.sh --report Csv
87-
```
77+
```shell
78+
./../../scripts/analysis/analyze.sh --report Csv
79+
```
8880
89-
- Jupyter notebook reports when Python and Conda are installed:
81+
- Jupyter notebook reports when Python and Conda are installed:
9082
91-
```shell
92-
./../../scripts/analysis/analyze.sh --report Jupyter
93-
```
83+
```shell
84+
./../../scripts/analysis/analyze.sh --report Jupyter
85+
```
9486
95-
- Graph visualizations when Node.js and npm are installed:
87+
- Graph visualizations when Node.js and npm are installed:
9688
97-
```shell
98-
./../../scripts/analysis/analyze.sh --report Jupyter
99-
```
89+
```shell
90+
./../../scripts/analysis/analyze.sh --report Jupyter
91+
```
10092
101-
- All reports with Python, Conda, Node.js and npm installed:
93+
- All reports with Python, Conda, Node.js and npm installed:
10294
103-
```shell
104-
./../../scripts/analysis/analyze.sh
105-
```
95+
```shell
96+
./../../scripts/analysis/analyze.sh
97+
```
10698
107-
- To explore the database yourself without any automatically generated reports and no additional requirements:
99+
- To explore the database yourself without any automatically generated reports and no additional requirements:
108100
109-
```shell
110-
./../../scripts/analysis/analyze.sh --explore
111-
```
101+
```shell
102+
./../../scripts/analysis/analyze.sh --explore
103+
```
104+
105+
👉 Open your browser and login to your [local Neo4j Web UI](http://localhost:7474/browser) with "neo4j" as user and the initial password you've chosen.
112106

113-
Then open your browser and login to your [local Neo4j Web UI](http://localhost:7474/browser) with "neo4j" as user and the initial password you've chosen.
107+
## GitHub Actions
114108

115-
👉 See [scripts/examples/analyzeAxonFramework.sh](./scripts/examples/analyzeAxonFramework.sh) as an example script that combines all the above steps for a Java Project.
116-
👉 See [scripts/examples/analyzeReactRouter.sh](./scripts/examples/analyzeReactRouter.sh) as an example script that combines all the above steps for a Typescript Project.
117-
👉 See [scripts/examples/analyzeAntDesign.sh](./scripts/examples/analyzeAntDesign.sh) as an example script that combines all the above steps for a large scale monorepo Typescript Project.
118109
👉 See [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.yml) on how to do this within a GitHub Actions Workflow.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
// Create index for the absolute file name
2+
3+
CREATE INDEX INDEX_ABSOLUTE_FILE_NAME IF NOT EXISTS FOR (file:File) ON (file.absoluteFileName)

cypher/Typescript_Enrichment/Link_projects_to_npm_packages.cypher

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ MATCH (npmPackage:NPM:Package)
2626
// that contains the package.json file
2727
SET npmPackage.relativeFileDirectory = ltrim(relativeNpmPackageDirectory, '/')
2828
,project.version = npmPackage.version
29+
,project.packageName = npmPackage.name
2930
RETURN count(*) AS numberOfCreatedNpmPackageRelationships
3031
// Detailed results for debugging
3132
//RETURN npmPackage.fileName AS npmPackageFileName

cypher/Typescript_Enrichment/Mark_test_modules.cypher

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ MATCH (m:Module)
44
WITH m
55
,(m.globalFqn contains '/__tests__/') OR
66
(m.globalFqn contains '/test/') OR
7+
(m.globalFqn contains '/testing/') OR
8+
(m.globalFqn contains '-tests/') OR
79
(m.globalFqn contains '/tests/') AS isInTestFolder
810
,(m.globalFqn contains '/__mocks__/') OR
911
(m.globalFqn contains '/mock/') OR

init.sh

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
3+
# Initializes a new analysis project by creating all necessary directories based on the given input parameter with the analysis name.
4+
5+
# Note: This script needs to be executed in the root of this directory (= same directory as this file)
6+
7+
# Fail on any error ("-e" = exit on first error, "-o pipefail" exist on errors within piped commands)
8+
set -o errexit -o pipefail
9+
10+
ARTIFACTS_DIRECTORY=${ARTIFACTS_DIRECTORY:-"artifacts"}
11+
SOURCE_DIRECTORY=${SOURCE_DIRECTORY:-"source"}
12+
13+
# Read the first (and only) parameter containing the name of the analysis.
14+
analysisName="${1}"
15+
if [ -z "${analysisName}" ]; then
16+
echo "init: Error: Missing parameter <analysisName>." >&2
17+
echo "init: Usage example: ${0} <analysisName>" >&2
18+
exit 1
19+
fi
20+
21+
nameOfThisScript=$(basename "${0}")
22+
if [ ! -f "./${nameOfThisScript}" ]; then
23+
echo "init: Error: Please execute the script in the root directory of the code-graph-analysis-pipeline repository." >&2
24+
echo "init: Change to the directory of this ${nameOfThisScript} script and execute it from there." >&2
25+
exit 1
26+
fi
27+
28+
# Check if initial password environment variable is set
29+
if [ -z "${NEO4J_INITIAL_PASSWORD}" ]; then
30+
echo "init: Error: Environment variable NEO4J_INITIAL_PASSWORD is recommended to be set first. Use 'export NEO4J_INITIAL_PASSWORD=<your-own-password>'."
31+
exit 1
32+
fi
33+
34+
# Create the temporary directory for all analysis projects if it hadn't been created yet.
35+
mkdir -p ./temp
36+
cd ./temp
37+
38+
# Create the analysis directory inside the temp directory using the given parameter if it hadn't been created yet.
39+
mkdir -p "./${analysisName}"
40+
cd "./${analysisName}"
41+
42+
# Create the artifacts directory inside the analysis directory for e.g. Java jar/ear files if it hadn't been created yet.
43+
mkdir -p "./${ARTIFACTS_DIRECTORY}"
44+
45+
# Create the source directory inside the analysis directory for source code projects/repositories if it hadn't been created yet.
46+
mkdir -p "./${SOURCE_DIRECTORY}"
47+
48+
# Create symbolic links to the most common scripts for code analysis.
49+
ln -s "./../../scripts/analysis/analyze.sh" .
50+
ln -s "./../../scripts/startNeo4j.sh" .
51+
ln -s "./../../scripts/stopNeo4j.sh" .
52+
53+
echo "init: Successfully initialized analysis project ${analysisName}" >&2

scripts/importGit.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,7 @@ postGitPluginImport() {
150150
execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_commit_sha.cypher"
151151
execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_file_name.cypher"
152152
execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_file_relative_path.cypher"
153+
execute_cypher "${GIT_LOG_CYPHER_DIR}/Index_absolute_file_name.cypher"
153154

154155
commonPostGitImport
155156

scripts/scanTypescript.sh

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -115,11 +115,15 @@ is_valid_scan_result() {
115115
}
116116

117117
is_change_detected() {
118-
# Scan and analyze Typescript sources only when they had been changed
118+
local COLOR_DARK_GREY='\033[0;30m'
119+
local COLOR_DEFAULT='\033[0m'
119120
local source_directory_name; source_directory_name=$(basename "${source_directory}");
121+
122+
echo -e "${COLOR_DARK_GREY}"
120123
changeDetectionHashFilePath="./${SOURCE_DIRECTORY}/typescriptScanChangeDetection-${source_directory_name}.sha"
121124
changeDetectionReturnCode=$( source "${SCRIPTS_DIR}/detectChangedFiles.sh" --readonly --hashfile "${changeDetectionHashFilePath}" --paths "${source_directory}")
122-
125+
echo -e "${COLOR_DEFAULT}"
126+
123127
if [ "${changeDetectionReturnCode}" == "0" ] && [ "${TYPESCRIPT_SCAN_CHANGE_DETECTION}" = true ]; then
124128
true
125129
else
@@ -143,6 +147,10 @@ total_source_directories=$(echo "${source_directories}" | wc -l | awk '{print $1
143147
processed_source_directories=0
144148

145149
for source_directory in ${source_directories}; do
150+
processed_source_directories=$((processed_source_directories + 1))
151+
progress_info_source_dirs="${processed_source_directories}/${total_source_directories}"
152+
153+
# Scan and analyze Typescript sources only when they had been changed
146154
if is_change_detected; then
147155
echo "scanTypescript: Files in ${source_directory} unchanged. Scan skipped."
148156
continue # skipping scan since it had already be done according to change detection.
@@ -151,9 +159,6 @@ for source_directory in ${source_directories}; do
151159
#Debugging log for change detection. "scan_directory" already logs scanning and the source directory.
152160
#echo "scanTypescript: Detected change (${changeDetectionReturnCode}) in ${source_directory}. Scanning Typescript source using @jqassistant/ts-lce."
153161

154-
processed_source_directories=$((processed_source_directories + 1))
155-
progress_info_source_dirs="${processed_source_directories}/${total_source_directories}"
156-
157162
if [ -f "${source_directory}/tsconfig.json" ] \
158163
&& scan_directory "${source_directory}" "${progress_info_source_dirs}" \
159164
&& is_valid_scan_result "${source_directory}"
@@ -164,14 +169,17 @@ for source_directory in ${source_directories}; do
164169

165170
echo "scanTypescript: Info: Unsuccessful or skipped source directory scan. Scan all contained packages individually." >&2
166171
contained_package_directories=$( find_directories_with_package_json_file "${source_directory}" )
167-
echo "scanTypescript: contained_package_directories:" >&2
168-
echo "${contained_package_directories}" >&2
172+
#Debugging: List all package directories.
173+
#echo "scanTypescript: contained_package_directories:" >&2
174+
#echo "${contained_package_directories}" >&2
169175
total_package_directories=$(echo "${contained_package_directories}" | wc -l | awk '{print $1}')
170176
processed_package_directories=0
171177

178+
main_source_directory_name=$(basename "${source_directory}");
179+
172180
for contained_package_directory in ${contained_package_directories}; do
173181
processed_package_directories=$((processed_package_directories + 1))
174-
progress_info_package_dirs="${progress_info_source_dirs}: ${processed_package_directories}/${total_package_directories}"
182+
progress_info_package_dirs="${main_source_directory_name} ${progress_info_source_dirs}: ${processed_package_directories}/${total_package_directories}"
175183
scan_directory "${contained_package_directory}" "${progress_info_package_dirs}"
176184
done
177185

0 commit comments

Comments
 (0)