Skip to content

Commit 049f591

Browse files
JohTJohT
committed
Add selectable settings profiles for analysis
1 parent 8930373 commit 049f591

File tree

5 files changed

+102
-35
lines changed

5 files changed

+102
-35
lines changed

‎scripts/SCRIPTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ Script | Directory | Description
1717
| [generateMarkdownReference.sh](./generateMarkdownReference.sh) | | Generates "REPORTS.md" containing a reference to all scripts in this directory and its subdirectories. |
1818
| [generateScriptReference.sh](./generateScriptReference.sh) | | Generates "SCRIPTS.md" containing a reference to all scripts in this directory and its subdirectories. |
1919
| [prepareAnalysis.sh](./prepareAnalysis.sh) | | Prepares and validates the graph database before analysis |
20+
| [Default.sh](./profiles/Default.sh) | profiles | Sets (if any) settings variables for a default analysis. |
21+
| [Neo4jv4.sh](./profiles/Neo4jv4.sh) | profiles | Sets all settings variables for an analysis with Neo4j v4.4.x (long term support (LTS) version as of may 2023). |
22+
| [Neo4jv5.sh](./profiles/Neo4jv5.sh) | profiles | Sets all settings variables for an analysis with Neo4j v5.x (newest version as of june 2023). |
2023
| [CentralityCsv.sh](./reports/CentralityCsv.sh) | reports | Looks for centrality using the Graph Data Science Library of Neo4j and creates CSV reports. |
2124
| [CommunityCsv.sh](./reports/CommunityCsv.sh) | reports | Detects communities using the Graph Data Science Library of Neo4j and creates CSV reports. |
2225
| [DatabaseCsvExport.sh](./reports/DatabaseCsvExport.sh) | reports | Exports the whole graph database as a CSV file using the APOC procedure "apoc.export.csv.all" |

‎scripts/analysis/analyze.sh‎

Lines changed: 45 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -10,47 +10,42 @@
1010
# It is recommended to create an empty directory (preferrable "temp")
1111
# and change into it prior to starting this script.
1212

13-
# Note: The first argument "--name" is requried. It is used to create the working directory
13+
# Note: The argument "--name" is requried. It is used to create the working directory
1414
# as well as to find the script "scripts/artifacts/download<name>.sh" to download the artifacts.
1515

16-
# Note: The second argument "--version" is optional.
16+
# Note: The argument "--version" is also required.
1717
# It is appended to the working directory to further distinguish the results.
18+
# The version is passed to the artifacts download script as an argument.
19+
20+
# Note: The argument "--report" is optional. The default value is "All".
21+
# It selects the report compilation, a named group of reports. Besides the default "All" there are e.g. "Csv" and "Jupyter".
22+
# This makes it possible to run only a subset of the reports. For example "Csv" won't need python to be set up and runs therefore much faster.
23+
# Implemented is this as a script in "scripts/reports/compilations" that starts with the report compilation name followed by "Reports.sh".
24+
25+
# Note: The argument "--profile" is optional. The default value is "Default".
26+
# It selects a settings profile that sets all suitable variables for the analysis.
27+
# This makes it possible to run an analysis with e.g. Neo4j v4 instead of v5. Further profiles might come in future.
28+
# Implemented is this as a script in "scripts/profiles" that starts with the settings profile name followed by ".sh".
1829

1930
# Note: The script and its sub scripts are designed to be as efficient as possible
2031
# when it comes to subsequent executions.
2132
# Existing downloads, installations, artifacts, scans and processes will be detected.
2233

23-
# Overrideable environment variables (optional, defaults also defined in sub scripts where needed)
24-
NEO4J_EDITION=${NEO4J_EDITION:-"community"} # Choose "community" or "enterprise"
25-
NEO4J_VERSION=${NEO4J_VERSION:-"5.9.0"}
26-
JQASSISTANT_CLI_VERSION=${JQASSISTANT_CLI_VERSION:-"2.0.3"} # 2.0.3 is the newest version (june 2023) compatible with Neo4j v5
27-
28-
# Overrideable environment variables for ports (optional, defaults also defined in sub scripts where needed)
29-
# Override them if you need to run multiple neo4j database servers in parallel.
30-
NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"}
31-
NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"}
32-
NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"}
33-
NEO4J_HTTP_TRANSACTION_ENDPOINT=${NEO4J_HTTP_TRANSACTION_ENDPOINT:-"db/neo4j/tx/commit"} # Neo4j v5: "db/<name>/tx/commit", Neo4j v4: "db/data/transaction/commit",
34-
35-
# Overrideable environment variables for the Neo4j APOC plugin (optional, defaults also defined in sub scripts where needed)
36-
# Override them if you want to use Neo4j v4 instead of v5
37-
NEO4J_APOC_PLUGIN_EDITION=${NEO4J_APOC_PLUGIN_EDITION:-"core"} #Awesome Procedures for Neo4j Plugin Edition (Neo4j v4.4.x "all", Neo4j >= v5 "core")
38-
NEO4J_APOC_PLUGIN_GITHUB=${NEO4J_APOC_PLUGIN_GITHUB:-"neo4j/apoc"} #Awesome Procedures for Neo4j Plugin GitHub User/Repository (Neo4j v4.4.x "neo4j-contrib/neo4j-apoc-procedures", Neo4j >= v5 "neo4j/apoc")
39-
40-
NEO4J_GDS_PLUGIN_VERSION=${NEO4J_GDS_PLUGIN_VERSION:-"2.4.0"} # Graph Data Science Plugin Version 2.4.x of is compatible with Neo4j 5.x
41-
34+
# Overrideable variables with directory names
4235
ARTIFACT_SCRIPTS_DIRECTORY=${ARTIFACT_SCRIPTS_DIRECTORY:-"artifacts"}
4336
REPORTS_SCRIPTS_DIRECTORY=${REPORTS_SCRIPTS_DIRECTORY:-"reports"}
4437
REPORT_COMPILATIONS_SCRIPTS_DIRECTORY=${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY:-"compilations"}
38+
SETTINGS_PROFILE_SCRIPTS_DIRECTORY=${SETTINGS_PROFILE_SCRIPTS_DIRECTORY:-"profiles"}
4539

4640
# Function to display script usage
4741
usage() {
48-
echo "Usage: $0 --name <name> --version <version> [--report <All (default), Csv, Jupyter,...>]"
42+
echo "Usage: $0 --name <name> --version <version> [--report <All (default), Csv, Jupyter,...>] [--profile <default, neo4jv5, neo4jv4,...>]"
4943
exit 1
5044
}
5145

5246
# Default values
5347
analysisReportCompilation="All"
48+
settingsProfile="default"
5449

5550
# Parse command line arguments
5651
while [[ $# -gt 0 ]]; do
@@ -68,8 +63,12 @@ while [[ $# -gt 0 ]]; do
6863
analysisReportCompilation="$2"
6964
shift
7065
;;
66+
--profile)
67+
settingsProfile="$2"
68+
shift
69+
;;
7170
*)
72-
echo "Error (analyze): Unknown option: ${key}"
71+
echo "analyze: Error: Unknown option: ${key}"
7372
usage
7473
;;
7574
esac
@@ -78,25 +77,31 @@ done
7877

7978
# Check if the name is provided
8079
if [[ -z ${analysisName} ]]; then
81-
echo "Error (analyze): Name is required."
80+
echo "analyze ${analysisName}: Error: Name is required."
8281
usage
8382
fi
8483

8584
# Check if the version is provided
8685
if [[ -z ${analysisVersion} ]]; then
87-
echo "Error (analyze): Version is required."
86+
echo "analyze ${analysisName}: Error: Version is required."
8887
usage
8988
fi
9089

9190
# Assure that the analysis name only consists of letters and numbers
9291
if ! [[ ${analysisName} =~ ^[[:alnum:]]+$ ]]; then
93-
echo "Error (analyze): Name can only contain letters and numbers."
92+
echo "analyze ${analysisName}: Error: Name can only contain letters and numbers."
9493
exit 1
9594
fi
9695

9796
# Assure that the analysis report compilation only consists of letters and numbers
9897
if ! [[ ${analysisReportCompilation} =~ ^[[:alnum:]]+$ ]]; then
99-
echo "Error (analyze): Report can only contain letters and numbers."
98+
echo "analyze ${analysisName}: Report can only contain letters and numbers."
99+
exit 1
100+
fi
101+
102+
# Assure that the settings profile only consists of letters and numbers
103+
if ! [[ ${settingsProfile} =~ ^[[:alnum:]]+$ ]]; then
104+
echo "analyze ${analysisName}: Error: Settings profile can only contain letters and numbers."
100105
exit 1
101106
fi
102107

@@ -111,26 +116,31 @@ echo "analyze ${analysisName}: ANALYSIS_SCRIPT_DIR=${ANALYSIS_SCRIPT_DIR}"
111116
SCRIPTS_DIR=${SCRIPTS_DIR:-$(dirname -- "${ANALYSIS_SCRIPT_DIR}")}
112117
echo "analyze ${analysisName}: SCRIPTS_DIR=${SCRIPTS_DIR}"
113118

114-
# Assure that there is a download script for the given analysis name.
115-
if [ ! -f "${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY}/download${analysisName}.sh" ] ; then
116-
echo "Error (analyze): No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
117-
exit 1
118-
fi
119-
120119
# Assure that there is a download script for the given analysis name argument.
121120
DOWNLOAD_SCRIPT="${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY}/download${analysisName}.sh"
122121
if [ ! -f "${DOWNLOAD_SCRIPT}" ] ; then
123-
echo "Error (analyze): No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
122+
echo "analyze ${analysisName}: Error: No download${analysisName}.sh script in the directory ${SCRIPTS_DIR}/${ARTIFACT_SCRIPTS_DIRECTORY} for analysis name ${analysisName}."
124123
exit 1
125124
fi
126125

127126
# Assure that there is a report compilation script for the given report argument.
128127
REPORT_COMPILATION_SCRIPT="${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY}/${analysisReportCompilation}Reports.sh"
129128
if [ ! -f "${REPORT_COMPILATION_SCRIPT}" ] ; then
130-
echo "Error (analyze): No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
129+
echo "analyze ${analysisName}: Error: No ${analysisReportCompilation}Reports.sh script in the directory ${SCRIPTS_DIR}/${REPORTS_SCRIPTS_DIRECTORY}/${REPORT_COMPILATIONS_SCRIPTS_DIRECTORY} for report name ${analysisReportCompilation}."
131130
exit 1
132131
fi
133132

133+
# Assure that there is a script file for the given settings profile name.
134+
SETTINGS_PROFILE_SCRIPT="${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY}/${settingsProfile}.sh"
135+
if [ ! -f "${SETTINGS_PROFILE_SCRIPT}" ] ; then
136+
echo "analyze ${analysisName}: Error: No ${settingsProfile}.sh script in the directory ${SCRIPTS_DIR}/${SETTINGS_PROFILE_SCRIPTS_DIRECTORY} for settings profile ${settingsProfile}."
137+
exit 1
138+
fi
139+
140+
# Execute the settings profile script that sets all the neccessary settings variables (overrideable by environment variables).
141+
echo "analyze ${analysisName}: Using analysis settings profile script ${SETTINGS_PROFILE_SCRIPT}"
142+
source "${SETTINGS_PROFILE_SCRIPT}" || exit 1
143+
134144
# Create working directory if it hadn't been created yet
135145
workingDirectory="${analysisName}-${analysisVersion}"
136146
mkdir -p "${workingDirectory}" || exit 2 # Create the working directory only if it doesn't exist

‎scripts/profiles/Default.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/usr/bin/env bash
2+
3+
# Sets (if any) settings variables for a default analysis.
4+
# The chosen settings are tested to be compatible and working.

‎scripts/profiles/Neo4jv4.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/usr/bin/env bash
2+
3+
# Sets all settings variables for an analysis with Neo4j v4.4.x (long term support (LTS) version as of may 2023).
4+
# The chosen settings are tested to be compatible and working.
5+
6+
NEO4J_VERSION=${NEO4J_VERSION:-"4.4.20"} # Version 4.4.x is the current long term support (LTS) version (may 2023)
7+
NEO4J_HTTP_TRANSACTION_ENDPOINT=${NEO4J_HTTP_TRANSACTION_ENDPOINT:-"db/data/transaction/commit"} # Since Neo4j v5 it is "db/<name>/tx/commit"
8+
9+
# Overrideable settings variables for ports (optional, defaults also defined in sub scripts where needed)
10+
# Override them if you need to run multiple neo4j database servers in parallel.
11+
NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"}
12+
NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"}
13+
NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"}
14+
15+
# Awesome Procedures (APOC) Plugin for Neo4j
16+
NEO4J_APOC_PLUGIN_VERSION=${NEO4J_APOC_PLUGIN_VERSION:-"4.4.0.15"} # Version number matches Neo4j version
17+
NEO4J_APOC_PLUGIN_EDITION=${NEO4J_APOC_PLUGIN_EDITION:-"all"} # Since Neo4j v5 only the core edition is maintained
18+
NEO4J_APOC_PLUGIN_GITHUB=${NEO4J_APOC_PLUGIN_GITHUB:-"neo4j-contrib/neo4j-apoc-procedures"} # Location for the old plugins compatible to Neo4j v4
19+
20+
NEO4J_GDS_PLUGIN_VERSION=${NEO4J_GDS_PLUGIN_VERSION:-"2.3.4"} # Graph Data Science Plugin Version 2.3.x is compatible with Neo4j 4.4.x
21+
22+
JQASSISTANT_CLI_VERSION=${JQASSISTANT_CLI_VERSION:-"1.12.2"} # Version 1.12.2 is the newest version (may 2023) compatible with Neo4j v4
23+
JQASSISTANT_CLI_ARTIFACT=${JQASSISTANT_CLI_ARTIFACT:-"jqassistant-commandline-neo4jv3"} # For Neo4j v3 & 4: "jqassistant-commandline-neo4jv3"
24+
JQASSISTANT_CLI_DISTRIBUTION=${JQASSISTANT_CLI_DISTRIBUTION:-"distribution.zip"} # Neo4j v3 & 4: "distribution.zip"
25+
JQASSISTANT_CONFIG_TEMPLATE=${JQASSISTANT_CONFIG_TEMPLATE:-"template-neo4jv4-jqassistant.yaml"}

‎scripts/profiles/Neo4jv5.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/usr/bin/env bash
2+
3+
# Sets all settings variables for an analysis with Neo4j v5.x (newest version as of june 2023).
4+
# The chosen settings are tested to be compatible and working.
5+
6+
NEO4J_VERSION=${NEO4J_VERSION:-"5.9.0"} # Version 5.9.0 is the current version of june 2023
7+
NEO4J_HTTP_TRANSACTION_ENDPOINT=${NEO4J_HTTP_TRANSACTION_ENDPOINT:-"db/neo4j/tx/commit"} # Since Neo4j v5 it is "db/<name>/tx/commit"
8+
9+
# Overrideable settings variables for ports (optional, defaults also defined in sub scripts where needed)
10+
# Override them if you need to run multiple neo4j database servers in parallel.
11+
NEO4J_HTTP_PORT=${NEO4J_HTTP_PORT:-"7474"}
12+
NEO4J_HTTPS_PORT=${NEO4J_HTTPS_PORT:-"7473"}
13+
NEO4J_BOLT_PORT=${NEO4J_BOLT_PORT:-"7687"}
14+
15+
# Awesome Procedures (APOC) Plugin for Neo4j
16+
NEO4J_APOC_PLUGIN_VERSION=${NEO4J_APOC_PLUGIN_VERSION:-"5.9.0"} # Version number matches Neo4j version since 5.x
17+
NEO4J_APOC_PLUGIN_EDITION=${NEO4J_APOC_PLUGIN_EDITION:-"core"} # Since Neo4j v5 the core edition is updated with Neo4j
18+
NEO4J_APOC_PLUGIN_GITHUB=${NEO4J_APOC_PLUGIN_GITHUB:-"neo4j/apoc"} # Core edition was moved to "neo4j/apoc" for Neo4j v5
19+
20+
NEO4J_GDS_PLUGIN_VERSION=${NEO4J_GDS_PLUGIN_VERSION:-"2.4.0"} # Version 2.4.0 is the newest version of june 2023 and compatible with Neo4j v5
21+
22+
JQASSISTANT_CLI_VERSION=${JQASSISTANT_CLI_VERSION:-"2.0.3"} # Version 2.0.3 is the newest version (june 2023) compatible with Neo4j v5
23+
JQASSISTANT_CLI_ARTIFACT=${JQASSISTANT_CLI_ARTIFACT:-"jqassistant-commandline-distribution"} # Since jQAssistant CLI v2: "jqassistant-commandline-distribution"
24+
JQASSISTANT_CLI_DISTRIBUTION=${JQASSISTANT_CLI_DISTRIBUTION:-"bin.zip"} # Since jQAssistant CLI v2: "bin.zip"
25+
JQASSISTANT_CONFIG_TEMPLATE=${JQASSISTANT_CONFIG_TEMPLATE:-"template-neo4jv5-jqassistant.yaml"}

0 commit comments

Comments
 (0)