Skip to content

Commit 0db7d44

Browse files
JohTJohT
committed
Optimize for larger code bases
1 parent 0c8ae21 commit 0db7d44

11 files changed

+100
-8
lines changed

‎cypher/CYPHER.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,23 @@ Script | Directory | Description
6767
| [Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher) | Community_Detection_for_Types | Community Detection for Types 6 Leiden Delete Labels |
6868
| [Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher) | Community_Detection_for_Types | Community Detection for Types 7 Add LeidenTypeCommunity+Id label to types with more than one member |
6969
| [Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher](./Community_Detection_for_Types/Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher) | Community_Detection_for_Types | Which type community spans several artifacts and how are the types distributed? |
70+
| [Count_nodes_and_relationships.cypher](./Count_nodes_and_relationships.cypher) | | Count nodes and relationships |
71+
| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher) | | Create a DEPENDS_ON relationship for every DEPENDS_ON_ARTIFACT |
72+
| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher) | | Create a DEPENDS_ON relationship for every DEPENDS_ON_PACKAGE |
73+
| [Create_index_for_full_qualified_type_name.cypher](./Create_index_for_full_qualified_type_name.cypher) | | Create index for full qualified type name |
7074
| [Cyclic_Dependencies.cypher](./Cyclic_Dependencies/Cyclic_Dependencies.cypher) | Cyclic_Dependencies | Cyclic Dependencies |
7175
| [Cyclic_Dependencies_Concatenated.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_Concatenated.cypher) | Cyclic_Dependencies | Cyclic Dependencies Concatenated |
7276
| [Cyclic_Dependencies_as_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as List |
7377
| [Cyclic_Dependencies_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as unwinded List |
78+
| [Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies between Artifacts as unwinded List |
7479
| [Export_the_whole_database_as_CSV.cypher](./Export_the_whole_database_as_CSV.cypher) | | Export the whole database as CSV |
7580
| [External_package_usage_overall.cypher](./External_Dependencies/External_package_usage_overall.cypher) | External_Dependencies | External package usage overall |
7681
| [External_package_usage_per_artifact.cypher](./External_Dependencies/External_package_usage_per_artifact.cypher) | External_Dependencies | External package usage per artifact |
7782
| [External_package_usage_per_artifact_and_package.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package.cypher) | External_Dependencies | External package usage per artifact and package |
83+
| [External_package_usage_per_artifact_and_package_without_annotations.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package_without_annotations.cypher) | External_Dependencies | External package usage per artifact and package without annotations Note: The exists operation for "isAnnotation" is inefficient for large graphs. |
7884
| [External_package_usage_per_type.cypher](./External_Dependencies/External_package_usage_per_type.cypher) | External_Dependencies | External package usage per type |
7985
| [External_package_usage_per_type_distribution.cypher](./External_Dependencies/External_package_usage_per_type_distribution.cypher) | External_Dependencies | External package usage per type distribution |
86+
| [External_package_usage_per_type_distribution_without_annotations.cypher](./External_Dependencies/External_package_usage_per_type_distribution_without_annotations.cypher) | External_Dependencies | External package usage per type distribution without annotations Note: The exists operation for "isAnnotation" is inefficient for large graphs. |
8087
| [External_types_per_artifact_using_requires.cypher](./External_Dependencies/External_types_per_artifact_using_requires.cypher) | External_Dependencies | External types per artifact using requires |
8188
| [Maven_POMs_and_their_declared_dependencies.cypher](./External_Dependencies/Maven_POMs_and_their_declared_dependencies.cypher) | External_Dependencies | Maven POMs and their declared dependencies |
8289
| [Extract_Custom_Manifest_Entries.cypher](./Extract_Custom_Manifest_Entries.cypher) | | Extract Custom Manifest Entries |

‎cypher/Create_index_for_full_qualified_type_name.cypher‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
// Create index for full qualified type name
2+
3+
// CREATE INDEX TYPE_FULL_QUALIFIED_NAME ON :Type(fqn)
4+
CREATE INDEX TYPE_FULL_QUALIFIED_NAME IF NOT EXISTS FOR (t:Type) ON (t.fqn)

‎cypher/External_Dependencies/External_package_usage_per_artifact_and_package.cypher‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@ UNWIND typeList AS type
1818
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
1919
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
2020
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
21-
,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
21+
// ,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
2222
WHERE isPrimitiveType = false
2323
AND isJavaType = false
2424
AND isAlsoInternalType = false
25-
AND isAnnotation = false
25+
// AND isAnnotation = false
2626
WITH numberOfTypesInPackage
2727
,artifactName
2828
,fullPackageName

‎cypher/External_Dependencies/External_package_usage_per_artifact_and_package_without_annotations.cypher‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// External package usage per artifact and package without annotations
2+
// Note: The exists operation for "isAnnotation" is inefficient for large graphs.
3+
4+
MATCH (artifact:Artifact)-[:CONTAINS]->(package:Package)
5+
MATCH (package)-[:CONTAINS]->(type:Type)
6+
WITH artifact, package, count(type) AS numberOfTypesInPackage, collect(type) as typeList
7+
UNWIND typeList AS type
8+
MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
9+
WHERE externalType.byteCodeVersion IS NULL
10+
WITH numberOfTypesInPackage
11+
,externalDependency
12+
,replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
13+
,package.fqn AS fullPackageName
14+
,package.name AS packageName
15+
,type.fqn AS fullTypeName
16+
,type.name AS typeName
17+
,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
18+
,externalType.name AS externalTypeName
19+
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
20+
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
21+
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
22+
,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
23+
WHERE isPrimitiveType = false
24+
AND isJavaType = false
25+
AND isAlsoInternalType = false
26+
AND isAnnotation = false
27+
WITH numberOfTypesInPackage
28+
,artifactName
29+
,fullPackageName
30+
,packageName
31+
,externalPackageName
32+
,count(externalDependency) AS numberOfExternalTypeCaller
33+
,sum(externalDependency.weight) AS numberOfExternalTypeCalls
34+
,collect(DISTINCT externalTypeName) AS externalTypeNames
35+
RETURN artifactName, fullPackageName
36+
,externalPackageName
37+
,numberOfExternalTypeCaller, numberOfExternalTypeCalls, numberOfTypesInPackage
38+
,externalTypeNames
39+
,packageName
40+
ORDER BY numberOfExternalTypeCaller DESC, artifactName ASC, fullPackageName ASC

‎cypher/External_Dependencies/External_package_usage_per_type_distribution.cypher‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,11 @@ UNWIND typeList AS type
1414
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
1515
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
1616
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
17-
,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
17+
// ,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
1818
WHERE isPrimitiveType = false
1919
AND isJavaType = false
2020
AND isAlsoInternalType = false
21-
AND isAnnotation = false
21+
// AND isAnnotation = false
2222
WITH artifactName
2323
,artifactTypes
2424
,fullTypeName

‎cypher/External_Dependencies/External_package_usage_per_type_distribution_without_annotations.cypher‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// External package usage per type distribution without annotations
2+
// Note: The exists operation for "isAnnotation" is inefficient for large graphs.
3+
4+
MATCH (artifact:Artifact)-[:CONTAINS]->(type:Type)
5+
WITH replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
6+
,count(type) AS artifactTypes
7+
,collect(type) AS typeList
8+
UNWIND typeList AS type
9+
MATCH (type)-[:DEPENDS_ON]->(externalType:Type)
10+
WHERE externalType.byteCodeVersion IS NULL
11+
WITH artifactName
12+
,artifactTypes
13+
,type.fqn AS fullTypeName
14+
,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
15+
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
16+
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
17+
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
18+
,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
19+
WHERE isPrimitiveType = false
20+
AND isJavaType = false
21+
AND isAlsoInternalType = false
22+
AND isAnnotation = false
23+
WITH artifactName
24+
,artifactTypes
25+
,fullTypeName
26+
,count(DISTINCT externalPackageName) AS numberOfExternalPackages
27+
WITH artifactName
28+
,artifactTypes
29+
,numberOfExternalPackages
30+
,count(DISTINCT fullTypeName) AS numberOfTypes
31+
,COLLECT(DISTINCT fullTypeName) AS nameOfTypes
32+
RETURN artifactName
33+
,artifactTypes
34+
,numberOfExternalPackages
35+
,numberOfTypes
36+
,100.0 / artifactTypes * numberOfTypes AS numberOfTypesPercentage
37+
,nameOfTypes
38+
ORDER BY artifactName ASC, numberOfExternalPackages ASC

‎scripts/executeJupyterNotebook.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ jupyter nbconvert --to notebook \
113113
--execute "${jupyter_notebook_file}" \
114114
--output "$jupyter_notebook_output_file_name" \
115115
--output-dir="./" \
116-
--ExecutePreprocessor.timeout=120 \
116+
--ExecutePreprocessor.timeout=240 \
117117
|| exit 5
118118

119119
# Convert the Jupyter Notebook to Markdown

‎scripts/executeQuery.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ error_message=$( echo "${cyper_query_result}" | jq -r '.errors[0] // empty' )
9595
if [[ -n "${error_message}" ]]; then
9696
redColor='\033[0;31m'
9797
noColor='\033[0m' # No Color
98-
echo -e "${redColor}${error_message}${noColor}" >&2
98+
echo -e "${redColor}${cypher_query_file_name}: ${error_message}${noColor}" >&2
9999
fi
100100

101101
# Output results in CSV format

‎scripts/prepareAnalysis.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ source "${SCRIPTS_DIR}/executeQueryFunctions.sh"
2828
PACKAGE_WEIGHTS_CYPHER_DIR="$CYPHER_DIR/Package_Relationship_Weights"
2929
PACKAGE_METRICS_CYPHER_DIR="$CYPHER_DIR/Metrics"
3030

31+
# Preparation - Create indizes
32+
execute_cypher "${CYPHER_DIR}/Create_index_for_full_qualified_type_name.cypher" || exit 1
33+
3134
# Preparation - Create DEPENDS_ON for every DEPENDS_ON_PACKAGE relationship
3235
execute_cypher_expect_results "${CYPHER_DIR}/Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher" || exit 1
3336
execute_cypher_expect_results "${CYPHER_DIR}/Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher" || exit 1

‎scripts/templates/template-neo4j-v4.conf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,4 @@ dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError
1919
dbms.memory.transaction.global_max_size=384m
2020

2121
# Memory: Limit the amount of memory that a single transaction can consume.
22-
dbms.memory.transaction.max_size=256m
22+
dbms.memory.transaction.max_size=512m

0 commit comments

Comments
 (0)