Skip to content

Commit b81c15d

Browse files
JohTJohT
committed
Provide generic download script for Typescript projects
1 parent 775a218 commit b81c15d

File tree

5 files changed

+225
-86
lines changed

5 files changed

+225
-86
lines changed

‎COMMANDS.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
- [Start Neo4j Graph Database](#start-neo4j-graph-database)
2525
- [Setup jQAssistant Java Code Analyzer](#setup-jqassistant-java-code-analyzer)
2626
- [Download Maven Artifacts to analyze](#download-maven-artifacts-to-analyze)
27+
- [Download Typescript project to analyze](#download-typescript-project-to-analyze)
2728
- [Reset the database and scan the java artifacts](#reset-the-database-and-scan-the-java-artifacts)
2829
- [Import git data](#import-git-data)
2930
- [Import aggregated git log](#import-aggregated-git-log)
@@ -221,6 +222,27 @@ to download a Maven artifact into the artifacts directory:
221222
- `-t <maven artifact type (optional, defaults to jar)>`
222223
- `-d <target directory for the downloaded file (optional, defaults to "artifacts")>`
223224

225+
### Download Typescript project to analyze
226+
227+
Use [downloadTypescriptProject.sh](./scripts/downloader/downloadTypescriptProject.sh) with the following options
228+
to download a Typescript project using git clone and prepare it for analysis:
229+
230+
- `--url` Git clone URL (required)
231+
- `--version` Version of the project
232+
- `--tag` Tag to switch to after "git clone" (optional, default = version)
233+
- `--project` Name of the project/repository (optional, default = clone url file name without .git extension)
234+
- `--packageManager` One of "npm", "pnpm" or "yarn". (optional, default = "npm")
235+
236+
Here is an example:
237+
238+
```shell
239+
./../../downloadTypescriptProject.sh \
240+
--url https://github.com/remix-run/react-router.git \
241+
--version 6.24.0 \
242+
--tag "[email protected]" \
243+
--packageManager pnpm
244+
```
245+
224246
### Reset the database and scan the java artifacts
225247

226248
Use [resetAndScan.sh](./scripts/resetAndScan.sh) to scan the local `artifacts` directory with the previously downloaded Java artifacts and write the data into the local Neo4J Graph database using jQAssistant. It also uses some jQAssistant "concepts" to

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -178,9 +178,9 @@ The [Code Structure Analysis Pipeline](./.github/workflows/java-code-analysis.ym
178178
👉 Create a new Jupyter report script in the [scripts/reports](./scripts/reports/) directory. Take [OverviewJupyter.sh](./scripts/reports/OverviewJupyter.sh) as a reference for example.
179179
👉 The script will automatically be included because of the directory and its name ending with "Jupyter.sh".
180180

181-
- How can i add another code basis to be analyzed automatically?
182-
👉 Create a new download script in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference.
183-
👉 Run the script separately before executing [analyze.sh](./scripts/analysis/analyze.sh) also in the [pipeline](./.github/workflows/java-code-analysis.yml).
181+
- How can i analyze a different code basis automatically?
182+
👉 Create a new download script like the ones in the [scripts/downloader](./scripts/downloader/) directory. Take for example [downloadAxonFramework.sh](./scripts/downloader/downloadAxonFramework.sh) as a reference for Java projects and [downloadReactRouter.sh](./scripts/downloader/downloadReactRouter.sh) as a reference for Typescript projects.
183+
👉 After downloading, run [analyze.sh](./scripts/analysis/analyze.sh). You can find these steps also in the [pipeline](./.github/workflows/java-code-analysis.yml) as a reference.
184184

185185
- How can i trigger a full re-scan of all artifacts?
186186
👉 Delete the file `artifactsChangeDetectionHash.txt` in the `artifacts` directory.

‎scripts/downloader/downloadAntDesign.sh‎

Lines changed: 14 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -4,53 +4,26 @@
44
# The source files are written into the "source" directory of the current analysis directory.
55
# After scanning it with jQAssistant Typescript Plugin the resulting JSON will be moved into the "artifacts" directory.
66

7-
# Note: The #-framed blocks are those that are specific to this download.
8-
# The other parts of the script can be reused/copied as a reference to write other download scripts.
9-
107
# Note: This script is meant to be started within the temporary analysis directory (e.g. "temp/AnalysisName/")
118

129
# Fail on any error (errexit = exit on first error, errtrace = error inherited from sub-shell ,pipefail exist on errors within piped commands)
1310
set -o errexit -o errtrace -o pipefail
1411

15-
# Get the analysis name from the middle part of the current file name (without prefix "download" and without extension)
16-
SCRIPT_FILE_NAME="$(basename -- "${BASH_SOURCE[0]}")"
17-
SCRIPT_FILE_NAME_WITHOUT_EXTENSION="${SCRIPT_FILE_NAME%%.*}"
18-
SCRIPT_FILE_NAME_WITHOUT_PREFIX_AND_EXTENSION="${SCRIPT_FILE_NAME_WITHOUT_EXTENSION##download}"
19-
ANALYSIS_NAME="${SCRIPT_FILE_NAME_WITHOUT_PREFIX_AND_EXTENSION}"
20-
SOURCE_DIRECTORY=${SOURCE_DIRECTORY:-"source"} # Get the source repository directory (defaults to "source")
21-
22-
echo "download${ANALYSIS_NAME}: SCRIPT_FILE_NAME=${SCRIPT_FILE_NAME}"
23-
echo "download${ANALYSIS_NAME}: SCRIPT_FILE_NAME_WITHOUT_EXTENSION=${SCRIPT_FILE_NAME_WITHOUT_EXTENSION}"
24-
echo "download${ANALYSIS_NAME}: ANALYSIS_NAME=${ANALYSIS_NAME}"
25-
2612
# Read the first input argument containing the version(s) of the artifact(s)
2713
if [ "$#" -ne 1 ]; then
28-
echo "Error (download${ANALYSIS_NAME}): Usage: $0 <version>" >&2
14+
echo "downloadAntDesign: Error: Usage: $0 <version>" >&2
2915
exit 1
3016
fi
31-
PROJECT_VERSION=$1
32-
echo "download${ANALYSIS_NAME}: PROJECT_VERSION=${PROJECT_VERSION}"
33-
34-
# Create runtime logs directory if it hasn't existed yet
35-
mkdir -p ./runtime/logs
36-
37-
################################################################
38-
# Download ant-design source files to be analyzed
39-
################################################################
40-
PROJECT_NAME="ant-design"
41-
FULL_PROJECT_NAME="${PROJECT_NAME}-${PROJECT_VERSION}"
42-
FULL_SOURCE_DIRECTORY="${SOURCE_DIRECTORY}/${FULL_PROJECT_NAME}"
43-
44-
if [ ! -d "${FULL_SOURCE_DIRECTORY}" ] ; then # only clone if source doesn't exist
45-
git clone --branch "${PROJECT_VERSION}" "https://github.com/ant-design/${PROJECT_NAME}.git" "${FULL_SOURCE_DIRECTORY}"
46-
fi
47-
(
48-
cd "${FULL_SOURCE_DIRECTORY}" || exit
49-
echo "download${ANALYSIS_NAME}: Installing dependencies..."
50-
npm install --verbose || exit
51-
echo "download${ANALYSIS_NAME}: Analyzing source..."
52-
npx --yes @jqassistant/ts-lce >"./../../runtime/logs/jqassistant-typescript-scan-${PROJECT_NAME}.log" 2>&1 || exit
53-
)
54-
mkdir -p artifacts/typescript
55-
mv -nv "${FULL_SOURCE_DIRECTORY}/.reports/jqa/ts-output.json" "artifacts/typescript/${FULL_PROJECT_NAME}.json"
56-
################################################################
17+
projectVersion="${1}"
18+
echo "downloadAntDesign: projectVersion=${projectVersion}"
19+
20+
## Get the directory of this script if not already set
21+
# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution.
22+
# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
23+
# This way non-standard tools like readlink aren't needed.
24+
DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )}
25+
echo "downloadReactRouter: DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR}"
26+
27+
source "${DOWNLOADER_SCRIPTS_DIR}/downloadTypescriptProject.sh" \
28+
--url https://github.com/ant-design/ant-design.git \
29+
--version "${projectVersion}"

‎scripts/downloader/downloadReactRouter.sh‎

Lines changed: 18 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -4,55 +4,31 @@
44
# The source files are written into the "source" directory of the current analysis directory.
55
# After scanning it with jQAssistant Typescript Plugin the resulting JSON will be moved into the "artifacts" directory.
66

7-
# Note: The #-framed blocks are those that are specific to this download.
8-
# The other parts of the script can be reused/copied as a reference to write other download scripts.
9-
107
# Note: This script is meant to be started within the temporary analysis directory (e.g. "temp/AnalysisName/")
8+
# Note: react-router uses pnpm as package manager which needs to be installed first
119

12-
# Note: react-router uses pnpm as package manager which needs to be installed
10+
# Requires downloadTypescriptProject.sh
1311

1412
# Fail on any error (errexit = exit on first error, errtrace = error inherited from sub-shell ,pipefail exist on errors within piped commands)
1513
set -o errexit -o errtrace -o pipefail
1614

17-
# Get the analysis name from the middle part of the current file name (without prefix "download" and without extension)
18-
SCRIPT_FILE_NAME="$(basename -- "${BASH_SOURCE[0]}")"
19-
SCRIPT_FILE_NAME_WITHOUT_EXTENSION="${SCRIPT_FILE_NAME%%.*}"
20-
SCRIPT_FILE_NAME_WITHOUT_PREFIX_AND_EXTENSION="${SCRIPT_FILE_NAME_WITHOUT_EXTENSION##download}"
21-
ANALYSIS_NAME="${SCRIPT_FILE_NAME_WITHOUT_PREFIX_AND_EXTENSION}"
22-
SOURCE_DIRECTORY=${SOURCE_DIRECTORY:-"source"} # Get the source repository directory (defaults to "source")
23-
24-
echo "download${ANALYSIS_NAME}: SCRIPT_FILE_NAME=${SCRIPT_FILE_NAME}"
25-
echo "download${ANALYSIS_NAME}: SCRIPT_FILE_NAME_WITHOUT_EXTENSION=${SCRIPT_FILE_NAME_WITHOUT_EXTENSION}"
26-
echo "download${ANALYSIS_NAME}: ANALYSIS_NAME=${ANALYSIS_NAME}"
27-
2815
# Read the first input argument containing the version(s) of the artifact(s)
2916
if [ "$#" -ne 1 ]; then
30-
echo "Error (download${ANALYSIS_NAME}): Usage: $0 <version>" >&2
17+
echo "downloadAntDesign: Error: Usage: $0 <version>" >&2
3118
exit 1
3219
fi
33-
PROJECT_VERSION=$1
34-
echo "download${ANALYSIS_NAME}: PROJECT_VERSION=${PROJECT_VERSION}"
35-
36-
# Create runtime logs directory if it hasn't existed yet
37-
mkdir -p ./runtime/logs
38-
39-
################################################################
40-
# Download react-router source files to be analyzed
41-
################################################################
42-
PROJECT_NAME="react-router"
43-
FULL_PROJECT_NAME="${PROJECT_NAME}-${PROJECT_VERSION}"
44-
FULL_SOURCE_DIRECTORY="${SOURCE_DIRECTORY}/${FULL_PROJECT_NAME}"
45-
46-
if [ ! -d "${FULL_SOURCE_DIRECTORY}" ] ; then # only clone if source doesn't exist
47-
git clone --branch "react-router@${PROJECT_VERSION}" "https://github.com/remix-run/${PROJECT_NAME}.git" "${FULL_SOURCE_DIRECTORY}"
48-
fi
49-
(
50-
cd "${FULL_SOURCE_DIRECTORY}" || exit
51-
echo "download${ANALYSIS_NAME}: Installing dependencies..."
52-
pnpm install --frozen-lockfile || exit
53-
echo "download${ANALYSIS_NAME}: Analyzing source..."
54-
npx --yes @jqassistant/ts-lce >"./../../runtime/logs/jqassistant-typescript-scan-${FULL_PROJECT_NAME}.log" 2>&1 || exit
55-
)
56-
mkdir -p artifacts/typescript
57-
mv -nv "${FULL_SOURCE_DIRECTORY}/.reports/jqa/ts-output.json" "artifacts/typescript/${FULL_PROJECT_NAME}.json"
58-
################################################################
20+
projectVersion="${1}"
21+
echo "downloadAntDesign: projectVersion=${projectVersion}"
22+
23+
## Get the directory of this script if not already set
24+
# Even if $BASH_SOURCE is made for Bourne-like shells it is also supported by others and therefore here the preferred solution.
25+
# CDPATH reduces the scope of the cd command to potentially prevent unintended directory changes.
26+
# This way non-standard tools like readlink aren't needed.
27+
DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR:-$( CDPATH=. cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P )}
28+
echo "downloadReactRouter: DOWNLOADER_SCRIPTS_DIR=${DOWNLOADER_SCRIPTS_DIR}"
29+
30+
source "${DOWNLOADER_SCRIPTS_DIR}/downloadTypescriptProject.sh" \
31+
--url https://github.com/remix-run/react-router.git \
32+
--version "${projectVersion}" \
33+
--tag "react-router@${projectVersion}" \
34+
--packageManager pnpm

‎scripts/downloader/downloadTypescriptProject.sh‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
#!/usr/bin/env bash
2+
3+
# Downloads the given version of a Typescript project from a git repository using git clone.
4+
# The cloned project is then moved into the "source" directory of the current analysis directory
5+
# where its dependencies are installed by the given package manager.
6+
# After scanning it with jQAssistant's Typescript Plugin, the resulting JSON will be moved into the "artifacts/typescript" directory.
7+
8+
# Command line options:
9+
# --url Git clone URL (required)
10+
# --version Version of the project
11+
# --tag Tag to switch to after "git clone" (optional, default = version)
12+
# --project Name of the project/repository (optional, default = clone url file name without .git extension)
13+
# --packageManager One of "npm", "pnpm" or "yarn". (optional, default = "npm")
14+
15+
# Note: This script is meant to be started within the temporary analysis directory (e.g. "temp/AnalysisName/")
16+
17+
# Fail on any error (errexit = exit on first error, errtrace = error inherited from sub-shell ,pipefail exist on errors within piped commands)
18+
set -o errexit -o errtrace -o pipefail
19+
20+
# Overrideable Defaults
21+
SOURCE_DIRECTORY=${SOURCE_DIRECTORY:-"source"}
22+
echo "downloadTypescriptProject: SOURCE_DIRECTORY=${SOURCE_DIRECTORY}"
23+
24+
# Display how this command is intended to be used including an example when wrong input parameters were detected
25+
usage() {
26+
echo ""
27+
echo "Usage: $0 \\"
28+
echo " --url <git-clone-url> \\"
29+
echo " --version <project version \\"
30+
echo " [ --tag <git-tag-for-that-version> (default=version) \\]"
31+
echo " [ --project <name-of-the-project> (default=url file name) \\]"
32+
echo " [ --packageManager <npm/pnpm/yarn> (default=npm) ]"
33+
echo "Example: $0 \\"
34+
echo " --url https://github.com/ant-design/ant-design.git \\"
35+
echo " --version 5.19.3"
36+
exit 1
37+
}
38+
39+
# Default command line option values
40+
cloneUrl=""
41+
projectName=""
42+
projectVersion=""
43+
projectTag=""
44+
packageManager="npm"
45+
46+
# Parse command line options
47+
while [[ $# -gt 0 ]]; do
48+
key="${1}"
49+
value="${2}"
50+
51+
case "${key}" in
52+
--url)
53+
cloneUrl="${value}"
54+
shift
55+
;;
56+
--project)
57+
projectName="${value}"
58+
shift
59+
;;
60+
--version)
61+
projectVersion="${value}"
62+
shift
63+
;;
64+
--tag)
65+
projectTag="${value}"
66+
shift
67+
;;
68+
--packageManager)
69+
packageManager="${value}"
70+
shift
71+
;;
72+
*)
73+
echo "downloadTypescriptProject Error: Unknown option: ${key}"
74+
usage
75+
;;
76+
esac
77+
shift
78+
done
79+
80+
if [[ -z ${cloneUrl} ]]; then
81+
echo "downloadTypescriptProject Error: Please specify an URL for git clone."
82+
usage
83+
fi
84+
85+
if ! curl --head --fail "${cloneUrl}" >/dev/null 2>&1; then
86+
echo "downloadTypescriptProject Error: Invalid URL: ${cloneUrl}"
87+
exit 1
88+
fi
89+
90+
if [[ -z ${projectName} ]]; then
91+
# When empty, infer the project name from the file name / last part of the clone url excluding the extension .git
92+
projectName=$(basename -s .git "${cloneUrl}")
93+
fi
94+
95+
if [[ -z ${projectVersion} ]]; then
96+
echo "downloadTypescriptProject Error: Please specify a project version."
97+
usage
98+
fi
99+
100+
if [[ -z ${projectTag} ]]; then
101+
# When empty, use the value of the option "version" as tag
102+
projectTag="${projectVersion}"
103+
fi
104+
105+
case "${packageManager}" in
106+
npm|pnpm|yarn)
107+
echo "downloadTypescriptProject Using package manager ${packageManager}"
108+
;;
109+
*)
110+
echo "downloadTypescriptProject Error: Unknown package manager: ${packageManager}"
111+
usage
112+
;;
113+
esac
114+
115+
if ! command -v "${packageManager}" &> /dev/null ; then
116+
echo "downloadTypescriptProject Error: Package manager ${packageManager} could not be found"
117+
exit 1
118+
fi
119+
120+
if ! command -v "npx" &> /dev/null ; then
121+
echo "downloadTypescriptProject Error: Command npx not found. It's needed to execute npm packages."
122+
exit 1
123+
fi
124+
125+
echo "downloadTypescriptProject: cloneUrl: ${cloneUrl}"
126+
echo "downloadTypescriptProject: projectName: ${projectName}"
127+
echo "downloadTypescriptProject: projectVersion: ${projectVersion}"
128+
echo "downloadTypescriptProject: projectTag: ${projectTag}"
129+
echo "downloadTypescriptProject: packageManager: ${packageManager}"
130+
131+
usePackageManagerToInstallDependencies() {
132+
echo "downloadTypescriptProject: Installing dependencies using ${packageManager}..."
133+
case "${packageManager}" in
134+
npm)
135+
# npm ci is not sufficient for projects like "ant-design" that rely on generating the package-lock
136+
# Even if this is not standard, this is an acceptable solution for standard and edge cases.
137+
npm install --ignore-scripts --verbose || exit
138+
;;
139+
pnpm)
140+
pnpm install --frozen-lockfile || exit
141+
;;
142+
yarn)
143+
yarn install --frozen-lockfile --ignore-scripts --non-interactive --verbose || exit
144+
;;
145+
esac
146+
}
147+
148+
# Create runtime logs directory if it hasn't existed yet
149+
mkdir -p ./runtime/logs
150+
151+
# Download the project to be analyzed
152+
fullProjectName="${projectName}-${projectVersion}"
153+
fullSourceDirectory="${SOURCE_DIRECTORY}/${fullProjectName}"
154+
155+
if [ ! -d "${fullSourceDirectory}" ] ; then # only clone if source doesn't exist
156+
echo "downloadTypescriptProject: Cloning ${cloneUrl} with version ${projectVersion}..."
157+
# A full clone is done since not only the source is scanned, but also the git log history.
158+
git clone --branch "${projectTag}" "${cloneUrl}" "${fullSourceDirectory}"
159+
fi
160+
(
161+
cd "${fullSourceDirectory}" || exit
162+
usePackageManagerToInstallDependencies
163+
echo "downloadTypescriptProject: Scanning Typescript source using @jqassistant/ts-lce..."
164+
npx --yes @jqassistant/ts-lce >"./../../runtime/logs/jqassistant-typescript-scan-${projectName}.log" 2>&1 || exit
165+
)
166+
echo "downloadTypescriptProject: Moving scanned results into the artifacts/typescript directory..."
167+
mkdir -p artifacts/typescript
168+
mv -nv "${fullSourceDirectory}/.reports/jqa/ts-output.json" "artifacts/typescript/${fullProjectName}.json"

0 commit comments

Comments
 (0)