Performance Tuning

.github/workflows/code-reports.yml

@@ -118,8 +118,20 @@ jobs:
       working-directory: temp
       run: ./../scripts/copyReportsIntoResults.sh
 
-    # Upload the results in case they are needed for troubleshooting for a couple of days
-    - name: Archive results
+    # Upload logs and unfinished reports in case of an error for troubleshooting
+    - name: Archive failed run with logs and unfinished results
+      if: failure()
+      uses: actions/upload-artifact@v3
+      with:
+        name: code-analysis-logs-java-${{ matrix.java }}-python-${{ matrix.python }}-mambaforge-${{ matrix.mambaforge }}
+        path: |
+          ./temp/**/runtime/*
+          ./results
+        retention-days: 5
+
+    # Upload successful results in case they are needed for troubleshooting
+    - name: Archive successful results
+      if: success()
       uses: actions/upload-artifact@v3
       with:
         name: code-report-results-java-${{ matrix.java }}-python-${{ matrix.python }}-mambaforge-${{ matrix.mambaforge }}

COMMANDS.md

@@ -64,6 +64,42 @@ a profile, the newest versions will be used. Profiles are scripts that can be fo
 - Use your own initial Neo4j password
 - For more details have a look at the script [analyze.sh](./scripts/analysis/analyze.sh)
 
+### Examples
+
+#### Start an analysis with CSV reports only
+
+If only the CSV reports are needed, that are the result of Cypher queries and don't need any further dependencies (like Python)
+the analysis can be speeded up with:
+
+```shell
+./../../scripts/analysis/analyze.sh --report Csv
+```
+
+#### Start an analysis with Jupyter reports only
+
+If only the Jupyter reports are needed e.g. when the CSV reports had already been generated, the this can be done with:
+
+```shell
+./../../scripts/analysis/analyze.sh --report Jupyter
+```
+
+#### Start an analysis without PDF generation
+
+Generating a PDF from a Jupyter notebook using [nbconvert](https://nbconvert.readthedocs.io) might take a while or even fail due to a timeout error. Here is an example on how to skip PDF generation:
+
+```shell
+SKIP_JUPYTER_NOTEBOOK_PDF_GENERATION=true ./../../scripts/analysis/analyze.sh
+```
+
+#### Setup everything to explore the graph manually
+
+To prepare everything for analysis including installation, configuration and preparation queries to explore the graph manually
+without report generation use this command:
+
+```shell
+./../../scripts/analysis/analyze.sh --explore
+```
+
 ## Generate Markdown References
 
 ### Update Cypher Reference

cypher/CYPHER.md

@@ -67,18 +67,29 @@ Script | Directory | Description
 | [Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_6_Leiden_Delete_Labels.cypher) | Community_Detection_for_Types | Community Detection for Types 6 Leiden Delete Labels |
 | [Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher](./Community_Detection_for_Types/Community_Detection_for_Types_7_Add_LeidenTypeCommunity_Id_label_to_types.cypher) | Community_Detection_for_Types | Community Detection for Types 7 Add LeidenTypeCommunity+Id label to types with more than one member |
 | [Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher](./Community_Detection_for_Types/Which_type_community_spans_several_artifacts_and_how_are_the_types_distributed.cypher) | Community_Detection_for_Types | Which type community spans several artifacts and how are the types distributed? |
+| [Count_nodes_and_relationships.cypher](./Count_nodes_and_relationships.cypher) |  | Count nodes and relationships |
+| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_ARTIFACT.cypher) |  | Create a DEPENDS_ON relationship for every DEPENDS_ON_ARTIFACT |
+| [Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher](./Create_a_DEPENDS_ON_relationship_for_every_DEPENDS_ON_PACKAGE.cypher) |  | Create a DEPENDS_ON relationship for every DEPENDS_ON_PACKAGE |
+| [Create_index_for_full_qualified_type_name.cypher](./Create_index_for_full_qualified_type_name.cypher) |  | Create index for the full qualified type name |
 | [Cyclic_Dependencies.cypher](./Cyclic_Dependencies/Cyclic_Dependencies.cypher) | Cyclic_Dependencies | Cyclic Dependencies |
 | [Cyclic_Dependencies_Concatenated.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_Concatenated.cypher) | Cyclic_Dependencies | Cyclic Dependencies Concatenated |
 | [Cyclic_Dependencies_as_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as List |
 | [Cyclic_Dependencies_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies as unwinded List |
+| [Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher](./Cyclic_Dependencies/Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher) | Cyclic_Dependencies | Cyclic Dependencies between Artifacts as unwinded List |
 | [Export_the_whole_database_as_CSV.cypher](./Export_the_whole_database_as_CSV.cypher) |  | Export the whole database as CSV |
-| [External_package_usage_overall.cypher](./External_Dependencies/External_package_usage_overall.cypher) | External_Dependencies | External package usage overall |
-| [External_package_usage_per_artifact.cypher](./External_Dependencies/External_package_usage_per_artifact.cypher) | External_Dependencies | External package usage per artifact |
+| [External_package_usage_overall.cypher](./External_Dependencies/External_package_usage_overall.cypher) | External_Dependencies | External package usage overall tuned |
+| [External_package_usage_per_artifact.cypher](./External_Dependencies/External_package_usage_per_artifact.cypher) | External_Dependencies | External package usage per artifact tuned |
 | [External_package_usage_per_artifact_and_package.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package.cypher) | External_Dependencies | External package usage per artifact and package |
-| [External_package_usage_per_type.cypher](./External_Dependencies/External_package_usage_per_type.cypher) | External_Dependencies | External package usage per type |
+| [External_package_usage_per_artifact_and_package_tuned.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package_tuned.cypher) | External_Dependencies | External package usage per artifact and package tuned |
+| [External_package_usage_per_artifact_and_package_without_annotations.cypher](./External_Dependencies/External_package_usage_per_artifact_and_package_without_annotations.cypher) | External_Dependencies | External package usage per artifact and package without annotations Note: The exists operation for "isAnnotation" is inefficient for large graphs. |
+| [External_package_usage_per_type.cypher](./External_Dependencies/External_package_usage_per_type.cypher) | External_Dependencies | External package usage per type tuned |
 | [External_package_usage_per_type_distribution.cypher](./External_Dependencies/External_package_usage_per_type_distribution.cypher) | External_Dependencies | External package usage per type distribution |
+| [External_package_usage_per_type_distribution_without_annotations.cypher](./External_Dependencies/External_package_usage_per_type_distribution_without_annotations.cypher) | External_Dependencies | External package usage per type distribution without annotations |
 | [External_types_per_artifact_using_requires.cypher](./External_Dependencies/External_types_per_artifact_using_requires.cypher) | External_Dependencies | External types per artifact using requires |
+| [Label_external_types_and_annotations.cypher](./External_Dependencies/Label_external_types_and_annotations.cypher) | External_Dependencies | Label external types and annotations |
+| [List_external_types_used.cypher](./External_Dependencies/List_external_types_used.cypher) | External_Dependencies | List external types used |
 | [Maven_POMs_and_their_declared_dependencies.cypher](./External_Dependencies/Maven_POMs_and_their_declared_dependencies.cypher) | External_Dependencies | Maven POMs and their declared dependencies |
+| [Remove_external_type_and_annotation_labels.cypher](./External_Dependencies/Remove_external_type_and_annotation_labels.cypher) | External_Dependencies | Remove external type and annotation labels |
 | [Extract_Custom_Manifest_Entries.cypher](./Extract_Custom_Manifest_Entries.cypher) |  | Extract Custom Manifest Entries |
 | [Get_Awesome_Procedures_On_Cypher_APOC_Version.cypher](./Get_Awesome_Procedures_On_Cypher_APOC_Version.cypher) |  | Get Awesome Procedures On Cypher APOC Version |
 | [Get_Graph_Data_Science_Library_Version.cypher](./Get_Graph_Data_Science_Library_Version.cypher) |  | Get Graph Data Science Library Version |

cypher/Candidates_for_Interface_Segregation.cypher

@@ -1,25 +1,25 @@
-// Candidates for Interface Segregation 
+// Candidates for Interface Segregation
 
-MATCH (type:Type)-[:DECLARES]->(method:Method)-[:INVOKES]->(dependentMethod:Method)<-[:DECLARES]-(dependentType:Type)
-MATCH (dependentType)-[:DECLARES]->(declaredMethod:Method)
+MATCH (type:Type)-[:DECLARES]->(method:Method)-[:INVOKES]->(dependentMethod:Method)
+MATCH (dependentMethod)<-[:DECLARES]-(dependentType:Type)
 MATCH (dependentType)-[:IMPLEMENTS*]->(superType:Type)-[:DECLARES]->(inheritedMethod:Method)
-WHERE type.fqn <> dependentType
+WHERE type.fqn <> dependentType.fqn
   AND dependentMethod.name IS NOT NULL
   AND inheritedMethod.name IS NOT NULL
   AND dependentMethod.name <> '<init>' // ignore constructors
   AND inheritedMethod.name <> '<init>' // ignore constructors
- WITH type
-     ,dependentType
+ WITH type.fqn                               AS fullTypeName
+     ,dependentType.fqn                      AS fullDependentTypeName
      ,collect(DISTINCT dependentMethod.name) AS calledMethodNames
-     ,count(DISTINCT dependentMethod) AS calledMethods
+     ,count(DISTINCT dependentMethod)        AS calledMethods
      // Count the different signatures without the return type
      // of all declared methods including the inherited ones
      ,count(DISTINCT split(method.signature, ' ')[1]) + count(DISTINCT split(inheritedMethod.signature, ' ')[1]) AS declaredMethods
 WHERE declaredMethods > calledMethods + 2
- WITH dependentType
+ WITH fullDependentTypeName
      ,declaredMethods
      ,calledMethodNames
      ,calledMethods
-     ,count(DISTINCT type.fqn) AS callerTypes
- RETURN dependentType.fqn, declaredMethods, calledMethodNames, calledMethods, callerTypes
- ORDER BY callerTypes DESC, declaredMethods DESC, dependentType.fqn
+     ,count(DISTINCT fullTypeName) AS callerTypes
+ RETURN fullDependentTypeName, declaredMethods, calledMethodNames, calledMethods, callerTypes
+ ORDER BY callerTypes DESC, declaredMethods DESC, fullDependentTypeName

cypher/Create_index_for_full_qualified_type_name.cypher

@@ -0,0 +1,3 @@
+// Create index for the full qualified type name
+
+CREATE INDEX INDEX_FULL_QUALIFIED_TYPE_NAME IF NOT EXISTS FOR (t:Type) ON (t.fqn)

cypher/Cyclic_Dependencies/Cyclic_Dependencies_as_List.cypher

@@ -2,6 +2,7 @@
 
 MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwardTarget:Type)<-[:CONTAINS]-(dependentPackage:Package)
 MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
+WHERE package <> dependentPackage
  WITH package
      ,dependentPackage
      ,collect(DISTINCT forwardSource.name  + '->' + forwardTarget.name)   AS forwardDependencies
@@ -13,8 +14,7 @@ MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(back
      ,size(forwardDependencies)  AS numberOfForwardDependencies
      ,size(backwardDependencies) AS numberOfBackwardDependencies
      ,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
-WHERE package <> dependentPackage
-  AND (size(forwardDependencies) > size(backwardDependencies)
+WHERE (size(forwardDependencies) > size(backwardDependencies)
    OR (size(forwardDependencies) = size(backwardDependencies)
   AND  size(package.fqn)    >= size(dependentPackage.fqn)))
 RETURN package.fqn          AS packageName

cypher/Cyclic_Dependencies/Cyclic_Dependencies_as_unwinded_List.cypher

@@ -2,6 +2,7 @@
 
 MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwardTarget:Type)<-[:CONTAINS]-(dependentPackage:Package)
 MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
+WHERE package <> dependentPackage
  WITH package
      ,dependentPackage
      ,collect(DISTINCT forwardSource.name  + '->' + forwardTarget.name)   AS forwardDependencies
@@ -13,8 +14,7 @@ MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(back
      ,size(forwardDependencies)  AS numberOfForwardDependencies
      ,size(backwardDependencies) AS numberOfBackwardDependencies
      ,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
-WHERE package <> dependentPackage
-  AND (size(forwardDependencies) > size(backwardDependencies)
+WHERE (size(forwardDependencies) > size(backwardDependencies)
    OR (size(forwardDependencies) = size(backwardDependencies)
   AND  size(package.fqn)    >= size(dependentPackage.fqn)))
 UNWIND (backwardDependencies + forwardDependencies) AS dependency

cypher/Cyclic_Dependencies/Cyclic_Dependencies_between_Artrifacts_as_unwinded_List.cypher

@@ -4,6 +4,8 @@ MATCH (package:Package)-[:CONTAINS]->(forwardSource:Type)-[:DEPENDS_ON]->(forwar
 MATCH (dependentPackage)-[:CONTAINS]->(backwardSource:Type)-[:DEPENDS_ON]->(backwardTarget:Type)<-[:CONTAINS]-(package)
 MATCH (artifact:Artifact)-[:CONTAINS]->(package)
 MATCH (dependentArtifact:Artifact)-[:CONTAINS]->(dependentPackage)
+WHERE artifact <> dependentArtifact
+  AND package  <> dependentPackage
  WITH artifact
      ,dependentArtifact
      ,package
@@ -19,9 +21,7 @@ MATCH (dependentArtifact:Artifact)-[:CONTAINS]->(dependentPackage)
      ,size(forwardDependencies)  AS numberOfForwardDependencies
      ,size(backwardDependencies) AS numberOfBackwardDependencies
      ,size(forwardDependencies) + size(backwardDependencies) AS numberOfAllCyclicDependencies
-WHERE artifact <> dependentArtifact
-  AND package  <> dependentPackage
-  AND (size(forwardDependencies) > size(backwardDependencies)
+WHERE (size(forwardDependencies) > size(backwardDependencies)
    OR (size(forwardDependencies) = size(backwardDependencies)
   AND  size(package.fqn)    >= size(dependentPackage.fqn)))
 UNWIND (backwardDependencies + forwardDependencies) AS dependency

cypher/External_Dependencies/External_package_usage_overall.cypher

@@ -3,27 +3,15 @@
  MATCH (type:Type)
   WITH count(type) as allTypes, collect(type) as typeList
 UNWIND typeList AS type
- MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
+ MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:ExternalType)
   WITH allTypes
       ,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
-      ,externalType.name                              AS externalTypeName
-      ,externalDependency
-      ,(NOT externalType.fqn CONTAINS '.')            AS isPrimitiveType
-      ,(externalType.fqn STARTS WITH 'java.')         AS isJavaType
-      ,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
-      ,(externalType.byteCodeVersion IS NULL)         AS isExternalType
- WHERE isPrimitiveType    = false
-   AND isJavaType         = false
-   AND isAlsoInternalType = false
-   AND isExternalType     = true
-  WITH allTypes
-      ,externalPackageName
-      ,count(externalDependency)          AS numberOfExternalTypeCaller
-      ,sum(externalDependency.weight)     AS numberOfExternalTypeCalls
-      ,collect(DISTINCT externalTypeName) AS externalTypeNames
+      ,count(externalDependency)                              AS numberOfExternalTypeCaller
+      ,sum(externalDependency.weight)                         AS numberOfExternalTypeCalls
+      ,collect(DISTINCT externalType.name)                    AS externalTypeNames
 RETURN externalPackageName
       ,numberOfExternalTypeCaller
       ,numberOfExternalTypeCalls
       ,allTypes
       ,externalTypeNames
- ORDER BY numberOfExternalTypeCaller DESC, externalTypeNames ASC
+ ORDER BY numberOfExternalTypeCaller DESC, externalPackageName ASC

cypher/External_Dependencies/External_package_usage_per_artifact.cypher

@@ -1,23 +1,18 @@
-// External package usage per artifact 
+// External package usage per artifact
 
- MATCH (artifact:Artifact)-[:CONTAINS]->(type:Type)
-  WITH artifact, count(type) as numberOfTypesInArtifact, collect(type) as typeList
-UNWIND typeList as type
- MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
- WHERE externalType.byteCodeVersion IS NULL
+ MATCH (artifact:Artifact:Archive)-[:CONTAINS]->(type:Type)
+  WITH replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
+      ,count(type)   AS numberOfTypesInArtifact
+      ,collect(type) AS typeList
+UNWIND typeList AS type
+ MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:ExternalType)
   WITH numberOfTypesInArtifact
       ,externalDependency
-      ,replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
+      ,artifactName
       ,type.fqn     AS fullTypeName
       ,type.name    AS typeName
       ,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
       ,externalType.name                              AS externalTypeName
-      ,(NOT externalType.fqn CONTAINS '.')            AS isPrimitiveType
-      ,(externalType.fqn STARTS WITH 'java.')         AS isJavaType
-      ,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
- WHERE isPrimitiveType    = false
-   AND isJavaType         = false
-   AND isAlsoInternalType = false
   WITH numberOfTypesInArtifact
       ,artifactName
       ,externalPackageName
@@ -30,4 +25,4 @@ RETURN artifactName
       ,numberOfExternalTypeCalls
       ,numberOfTypesInArtifact
       ,externalTypeNames
-ORDER BY artifactName ASC, numberOfExternalTypeCaller DESC
+ORDER BY artifactName ASC, numberOfExternalTypeCaller DESC, externalPackageName ASC

cypher/External_Dependencies/External_package_usage_per_artifact_and_package.cypher

@@ -1,28 +1,24 @@
-// External package usage per artifact and package 
+// External package usage per artifact and package
 
  MATCH (artifact:Artifact)-[:CONTAINS]->(package:Package)
  MATCH (package)-[:CONTAINS]->(type:Type)
-  WITH artifact, package, count(type) AS numberOfTypesInPackage, collect(type) as typeList
+  WITH replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
+      ,package.fqn   AS fullPackageName
+      ,package.name  AS packageName
+      ,count(type)   AS numberOfTypesInPackage
+      ,collect(type) AS typeList
 UNWIND typeList AS type
- MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:Type)
- WHERE externalType.byteCodeVersion IS NULL
+ MATCH (type)-[externalDependency:DEPENDS_ON]->(externalType:ExternalType)
+ WHERE NOT externalType:ExternalAnnotation
   WITH numberOfTypesInPackage
       ,externalDependency
-      ,replace(last(split(artifact.fileName, '/')), '.jar', '') AS artifactName
-      ,package.fqn  AS fullPackageName
-      ,package.name AS packageName
+      ,artifactName
+      ,fullPackageName
+      ,packageName
       ,type.fqn     AS fullTypeName
       ,type.name    AS typeName
       ,replace(externalType.fqn, '.' + externalType.name, '') AS externalPackageName
       ,externalType.name                              AS externalTypeName
-      ,(NOT externalType.fqn CONTAINS '.')            AS isPrimitiveType
-      ,(externalType.fqn STARTS WITH 'java.')         AS isJavaType
-      ,exists((externalType)-[:RESOLVES_TO]->(:Type)) AS isAlsoInternalType
-      ,exists((externalType)<-[:OF_TYPE]-()<-[:ANNOTATED_BY]-()) AS isAnnotation
- WHERE isPrimitiveType    = false
-   AND isJavaType         = false
-   AND isAlsoInternalType = false
-   AND isAnnotation       = false
   WITH numberOfTypesInPackage
       ,artifactName
       ,fullPackageName