Skip to content

Commit d672057

Browse files
JohTJohT
authored
Merge pull request #19 from JohT/feature/performance-tuning
Performance Tuning
2 parents 5b67e79 + 8a43979 commit d672057

25 files changed

+227
-103
lines changed

‎.github/workflows/code-reports.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,8 +118,20 @@ jobs:
118118
working-directory: temp
119119
run: ./../scripts/copyReportsIntoResults.sh
120120

121-
# Upload the results in case they are needed for troubleshooting for a couple of days
122-
- name: Archive results
121+
# Upload logs and unfinished reports in case of an error for troubleshooting
122+
- name: Archive failed run with logs and unfinished results
123+
if: failure()
124+
uses: actions/upload-artifact@v3
125+
with:
126+
name: code-analysis-logs-java-${{ matrix.java }}-python-${{ matrix.python }}-mambaforge-${{ matrix.mambaforge }}
127+
path: |
128+
./temp/**/runtime/*
129+
./results
130+
retention-days: 5
131+
132+
# Upload successful results in case they are needed for troubleshooting
133+
- name: Archive successful results
134+
if: success()
123135
uses: actions/upload-artifact@v3
124136
with:
125137
name: code-report-results-java-${{ matrix.java }}-python-${{ matrix.python }}-mambaforge-${{ matrix.mambaforge }}

‎COMMANDS.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,42 @@ a profile, the newest versions will be used. Profiles are scripts that can be fo
6464
- Use your own initial Neo4j password
6565
- For more details have a look at the script [analyze.sh](./scripts/analysis/analyze.sh)
6666

67+
### Examples
68+
69+
#### Start an analysis with CSV reports only
70+
71+
If only the CSV reports are needed, that are the result of Cypher queries and don't need any further dependencies (like Python)
72+
the analysis can be speeded up with:
73+
74+
```shell
75+
./../../scripts/analysis/analyze.sh --report Csv
76+
```
77+
78+
#### Start an analysis with Jupyter reports only
79+
80+
If only the Jupyter reports are needed e.g. when the CSV reports had already been generated, the this can be done with:
81+
82+
```shell
83+
./../../scripts/analysis/analyze.sh --report Jupyter
84+
```
85+
86+
#### Start an analysis without PDF generation
87+
88+
Generating a PDF from a Jupyter notebook using [nbconvert](https://nbconvert.readthedocs.io) might take a while or even fail due to a timeout error. Here is an example on how to skip PDF generation:
89+
90+
```shell
91+
SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
92+
```
93+
94+
#### Setup everything to explore the graph manually
95+
96+
To prepare everything for analysis including installation, configuration and preparation queries to explore the graph manually
97+
without report generation use this command:
98+
99+
```shell
100+
./../../scripts/analysis/analyze.sh --explore
101+
```
102+
67103
## Generate Markdown References
68104
69105
### Update Cypher Reference

‎cypher/CYPHER.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,18 +67,29 @@ Script | Directory | Description
6767
| [Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher) | Community_Detection_for_Types | Community Detection for Types 6 Leiden Delete Labels |
6868
| [Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher) | Community_Detection_for_Types | Community Detection for Types 7 Add LeidenTypeCommunity+Id label to types with more than one member |
6969
| [Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher](./Community_Detection_for_Types/Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher) | Community_Detection_for_Types | Which type community spans several artifacts and how are the types distributed? |
70+
| [Count_nodes_and_relationships.cypher](./Count_nodes_and_relationships.cypher) | | Count nodes and relationships |
71+
| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher) | | Create a DEPENDS_ON relationship for every DEPENDS_ON_ARTIFACT |
72+
| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher) | | Create a DEPENDS_ON relationship for every DEPENDS_ON_PACKAGE |
73+
| [Create_index_for_full_qualified_type_name.cypher](./Create_index_for_full_qualified_type_name.cypher) | | Create index for the full qualified type name |
7074
| [Cyclic_Dependencies.cypher](./Cyclic_Dependencies/Cyclic_Dependencies.cypher) | Cyclic_Dependencies | Cyclic Dependencies |
7175
| [Cyclic_Dependencies_Concatenated.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_Concatenated.cypher) | Cyclic_Dependencies | Cyclic Dependencies Concatenated |
7276
| [Cyclic_Dependencies_as_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as List |
7377
| [Cyclic_Dependencies_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as unwinded List |
78+
| [Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies between Artifacts as unwinded List |
7479
| [Export_the_whole_database_as_CSV.cypher](./Export_the_whole_database_as_CSV.cypher) | | Export the whole database as CSV |
75-
| [External_package_usage_overall.cypher](./External_Dependencies/External_package_usage_overall.cypher) | External_Dependencies | External package usage overall |
76-
| [External_package_usage_per_artifact.cypher](./External_Dependencies/External_package_usage_per_artifact.cypher) | External_Dependencies | External package usage per artifact |
80+
| [External_package_usage_overall.cypher](./External_Dependencies/External_package_usage_overall.cypher) | External_Dependencies | External package usage overall tuned |
81+
| [External_package_usage_per_artifact.cypher](./External_Dependencies/External_package_usage_per_artifact.cypher) | External_Dependencies | External package usage per artifact tuned |
7782
| [External_package_usage_per_artifact_and_package.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package.cypher) | External_Dependencies | External package usage per artifact and package |
78-
| [External_package_usage_per_type.cypher](./External_Dependencies/External_package_usage_per_type.cypher) | External_Dependencies | External package usage per type |
83+
| [External_package_usage_per_artifact_and_package_tuned.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package_tuned.cypher) | External_Dependencies | External package usage per artifact and package tuned |
84+
| [External_package_usage_per_artifact_and_package_without_annotations.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package_without_annotations.cypher) | External_Dependencies | External package usage per artifact and package without annotations Note: The exists operation for "isAnnotation" is inefficient for large graphs. |
85+
| [External_package_usage_per_type.cypher](./External_Dependencies/External_package_usage_per_type.cypher) | External_Dependencies | External package usage per type tuned |
7986
| [External_package_usage_per_type_distribution.cypher](./External_Dependencies/External_package_usage_per_type_distribution.cypher) | External_Dependencies | External package usage per type distribution |
87+
| [External_package_usage_per_type_distribution_without_annotations.cypher](./External_Dependencies/External_package_usage_per_type_distribution_without_annotations.cypher) | External_Dependencies | External package usage per type distribution without annotations |
8088
| [External_types_per_artifact_using_requires.cypher](./External_Dependencies/External_types_per_artifact_using_requires.cypher) | External_Dependencies | External types per artifact using requires |
89+
| [Label_external_types_and_annotations.cypher](./External_Dependencies/Label_external_types_and_annotations.cypher) | External_Dependencies | Label external types and annotations |
90+
| [List_external_types_used.cypher](./External_Dependencies/List_external_types_used.cypher) | External_Dependencies | List external types used |
8191
| [Maven_POMs_and_their_declared_dependencies.cypher](./External_Dependencies/Maven_POMs_and_their_declared_dependencies.cypher) | External_Dependencies | Maven POMs and their declared dependencies |
92+
| [Remove_external_type_and_annotation_labels.cypher](./External_Dependencies/Remove_external_type_and_annotation_labels.cypher) | External_Dependencies | Remove external type and annotation labels |
8293
| [Extract_Custom_Manifest_Entries.cypher](./Extract_Custom_Manifest_Entries.cypher) | | Extract Custom Manifest Entries |
8394
| [Get_Awesome_Procedures_On_Cypher_APOC_Version.cypher](./Get_Awesome_Procedures_On_Cypher_APOC_Version.cypher) | | Get Awesome Procedures On Cypher APOC Version |
8495
| [Get_Graph_Data_Science_Library_Version.cypher](./Get_Graph_Data_Science_Library_Version.cypher) | | Get Graph Data Science Library Version |

‎cypher/Candidates_for_Interface_Segregation.cypher‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,25 @@
1-
// Candidates for Interface Segregation
1+
// Candidates for Interface Segregation
22

3-
MATCH (type:Type)-[:DECLARES]->(method:Method)-[:INVOKES]->(dependentMethod:Method)<-[:DECLARES]-(dependentType:Type)
4-
MATCH (dependentType)-[:DECLARES]->(declaredMethod:Method)
3+
MATCH (type:Type)-[:DECLARES]->(method:Method)-[:INVOKES]->(dependentMethod:Method)
4+
MATCH (dependentMethod)<-[:DECLARES]-(dependentType:Type)
55
MATCH (dependentType)-[:IMPLEMENTS*]->(superType:Type)-[:DECLARES]->(inheritedMethod:Method)
6-
WHERE type.fqn <> dependentType
6+
WHERE type.fqn <> dependentType.fqn
77
AND dependentMethod.name IS NOT NULL
88
AND inheritedMethod.name IS NOT NULL
99
AND dependentMethod.name <> '<init>' // ignore constructors
1010
AND inheritedMethod.name <> '<init>' // ignore constructors
11-
WITH type
12-
,dependentType
11+
WITH type.fqn AS fullTypeName
12+
,dependentType.fqn AS fullDependentTypeName
1313
,collect(DISTINCT dependentMethod.name) AS calledMethodNames
14-
,count(DISTINCT dependentMethod) AS calledMethods
14+
,count(DISTINCT dependentMethod) AS calledMethods
1515
// Count the different signatures without the return type
1616
// of all declared methods including the inherited ones
1717
,count(DISTINCT split(method.signature, ' ')[1]) + count(DISTINCT split(inheritedMethod.signature, ' ')[1]) AS declaredMethods
1818
WHERE declaredMethods > calledMethods + 2
19-
WITH dependentType
19+
WITH fullDependentTypeName
2020
,declaredMethods
2121
,calledMethodNames
2222
,calledMethods
23-
,count(DISTINCT type.fqn) AS callerTypes
24-
RETURN dependentType.fqn, declaredMethods, calledMethodNames, calledMethods, callerTypes
25-
ORDER BY callerTypes DESC, declaredMethods DESC, dependentType.fqn
23+
,count(DISTINCT fullTypeName) AS callerTypes
24+
RETURN fullDependentTypeName, declaredMethods, calledMethodNames, calledMethods, callerTypes
25+
ORDER BY callerTypes DESC, declaredMethods DESC, fullDependentTypeName

‎cypher/Create_index_for_full_qualified_type_name.cypher‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
// Create index for the full qualified type name
2+
3+
CREATE INDEX INDEX_FULL_QUALIFIED_TYPE_NAME IF NOT EXISTS FOR (t:Type) ON (t.fqn)

‎cypher/Cyclic_Dependencies/Cyclic_Dependencies_as_List.cypher‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwardTarget:Type)<-[:CONTAINS]-(dependentPackage:Package)
44
MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
5+
WHERE package <> dependentPackage
56
WITH package
67
,dependentPackage
78
,collect(DISTINCT forwardSource.name + '->' + forwardTarget.name) AS forwardDependencies
@@ -13,8 +14,7 @@ MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(back
1314
,size(forwardDependencies) AS numberOfForwardDependencies
1415
,size(backwardDependencies) AS numberOfBackwardDependencies
1516
,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
16-
WHERE package <> dependentPackage
17-
AND (size(forwardDependencies) > size(backwardDependencies)
17+
WHERE (size(forwardDependencies) > size(backwardDependencies)
1818
OR (size(forwardDependencies) = size(backwardDependencies)
1919
AND size(package.fqn) >= size(dependentPackage.fqn)))
2020
RETURN package.fqn AS packageName

‎cypher/Cyclic_Dependencies/Cyclic_Dependencies_as_unwinded_List.cypher‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwardTarget:Type)<-[:CONTAINS]-(dependentPackage:Package)
44
MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
5+
WHERE package <> dependentPackage
56
WITH package
67
,dependentPackage
78
,collect(DISTINCT forwardSource.name + '->' + forwardTarget.name) AS forwardDependencies
@@ -13,8 +14,7 @@ MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(back
1314
,size(forwardDependencies) AS numberOfForwardDependencies
1415
,size(backwardDependencies) AS numberOfBackwardDependencies
1516
,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
16-
WHERE package <> dependentPackage
17-
AND (size(forwardDependencies) > size(backwardDependencies)
17+
WHERE (size(forwardDependencies) > size(backwardDependencies)
1818
OR (size(forwardDependencies) = size(backwardDependencies)
1919
AND size(package.fqn) >= size(dependentPackage.fqn)))
2020
UNWIND (backwardDependencies + forwardDependencies) AS dependency

‎cypher/Cyclic_Dependencies/Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwar
44
MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
55
MATCH (artifact:Artifact)-[:CONTAINS]->(package)
66
MATCH (dependentArtifact:Artifact)-[:CONTAINS]->(dependentPackage)
7+
WHERE artifact <> dependentArtifact
8+
AND package <> dependentPackage
79
WITH artifact
810
,dependentArtifact
911
,package
@@ -19,9 +21,7 @@ MATCH (dependentArtifact:Artifact)-[:CONTAINS]->(dependentPackage)
1921
,size(forwardDependencies) AS numberOfForwardDependencies
2022
,size(backwardDependencies) AS numberOfBackwardDependencies
2123
,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
22-
WHERE artifact <> dependentArtifact
23-
AND package <> dependentPackage
24-
AND (size(forwardDependencies) > size(backwardDependencies)
24+
WHERE (size(forwardDependencies) > size(backwardDependencies)
2525
OR (size(forwardDependencies) = size(backwardDependencies)
2626
AND size(package.fqn) >= size(dependentPackage.fqn)))
2727
UNWIND (backwardDependencies + forwardDependencies) AS dependency

‎cypher/External_Dependencies/External_package_usage_overall.cypher‎

Lines changed: 5 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,27 +3,15 @@
33
MATCH (type:Type)
44
WITH count(type) as allTypes, collect(type) as typeList
55
UNWIND typeList AS type
6-
MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
6+
MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:ExternalType)
77
WITH allTypes
88
,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
9-
,externalType.name AS externalTypeName
10-
,externalDependency
11-
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
12-
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
13-
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
14-
,(externalType.byteCodeVersion IS NULL) AS isExternalType
15-
WHERE isPrimitiveType = false
16-
AND isJavaType = false
17-
AND isAlsoInternalType = false
18-
AND isExternalType = true
19-
WITH allTypes
20-
,externalPackageName
21-
,count(externalDependency) AS numberOfExternalTypeCaller
22-
,sum(externalDependency.weight) AS numberOfExternalTypeCalls
23-
,collect(DISTINCT externalTypeName) AS externalTypeNames
9+
,count(externalDependency) AS numberOfExternalTypeCaller
10+
,sum(externalDependency.weight) AS numberOfExternalTypeCalls
11+
,collect(DISTINCT externalType.name) AS externalTypeNames
2412
RETURN externalPackageName
2513
,numberOfExternalTypeCaller
2614
,numberOfExternalTypeCalls
2715
,allTypes
2816
,externalTypeNames
29-
ORDER BY numberOfExternalTypeCaller DESC, externalTypeNames ASC
17+
ORDER BY numberOfExternalTypeCaller DESC, externalPackageName ASC

‎cypher/External_Dependencies/External_package_usage_per_artifact.cypher‎

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,18 @@
1-
// External package usage per artifact
1+
// External package usage per artifact
22

3-
MATCH (artifact:Artifact)-[:CONTAINS]->(type:Type)
4-
WITH artifact, count(type) as numberOfTypesInArtifact, collect(type) as typeList
5-
UNWIND typeList as type
6-
MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
7-
WHERE externalType.byteCodeVersion IS NULL
3+
MATCH (artifact:Artifact:Archive)-[:CONTAINS]->(type:Type)
4+
WITH replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
5+
,count(type) AS numberOfTypesInArtifact
6+
,collect(type) AS typeList
7+
UNWIND typeList AS type
8+
MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:ExternalType)
89
WITH numberOfTypesInArtifact
910
,externalDependency
10-
,replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
11+
,artifactName
1112
,type.fqn AS fullTypeName
1213
,type.name AS typeName
1314
,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
1415
,externalType.name AS externalTypeName
15-
,(NOT externalType.fqn CONTAINS '.') AS isPrimitiveType
16-
,(externalType.fqn STARTS WITH 'java.') AS isJavaType
17-
,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
18-
WHERE isPrimitiveType = false
19-
AND isJavaType = false
20-
AND isAlsoInternalType = false
2116
WITH numberOfTypesInArtifact
2217
,artifactName
2318
,externalPackageName
@@ -30,4 +25,4 @@ RETURN artifactName
3025
,numberOfExternalTypeCalls
3126
,numberOfTypesInArtifact
3227
,externalTypeNames
33-
ORDER BY artifactName ASC, numberOfExternalTypeCaller DESC
28+
ORDER BY artifactName ASC, numberOfExternalTypeCaller DESC, externalPackageName ASC

0 commit comments

Comments
 (0)