Skip to content

Optimize centrality reports using in-memory mutate #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Oct 14, 2023
Merged

Optimize centrality reports using in-memory mutate #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5f444a8
Optimize centrality reports using in-memory mutate
JohT Oct 9, 2023
22b1486
Fix command line input error for explore mode
JohT Oct 14, 2023
0215a88
Support empty=no relationship weight parameter
JohT Oct 14, 2023
8b36795
Add centrality for methods
JohT Oct 14, 2023
8376ed4
Update open graph data science to v2.5.0
JohT Oct 14, 2023
f68d1e6
Raise transaction memory to 3 GB
JohT Oct 14, 2023
fa12929
Write Modularity only for Artifacts & Packages
JohT Oct 14, 2023
2fa1342
Rename "without-empty" to "cleaned"
JohT Oct 14, 2023
44dfc6f
Add centrality summary containing all values
JohT Oct 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions cypher/Centrality/Centrality_10_Summary.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Centrality Summary

MATCH (codeUnit)
WHERE (codeUnit.incomingDependencies > 0 OR codeUnit.outgoingDependencies > 0)
AND $dependencies_projection_node IN LABELS(codeUnit)
RETURN coalesce(codeUnit.fqn, codeUnit.fileName, codeUnit.signature, codeUnit.name) AS name
,coalesce(codeUnit.name, replace(last(split(codeUnit.fileName, '/')), '.jar', '')) AS shortName
,codeUnit.centralityPageRank AS pageRank
,codeUnit.centralityArticleRank AS articleRank
,codeUnit.centralityBetweenness AS betweenness
,codeUnit.centralityCostEffectiveLazyForward AS costEffectiveLazyForward
,codeUnit.centralityHarmonic AS harmonicCloseness
,codeUnit.centralityCloseness AS closeness
,codeUnit.centralityHyperlinkInducedTopicSearchAuthority AS hyperlinkInducedTopicSearchAuthority
,codeUnit.centralityHyperlinkInducedTopicSearchHub AS hyperlinkInducedTopicSearchHub
,codeUnit.incomingDependencies AS incomingDependencies
,codeUnit.outgoingDependencies AS outgoingDependencies
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_2a_Page_Rank_Estimate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
//Centrality 2a Page Rank Estimate Memory
// Centrality 2a Page Rank Estimate Memory

CALL gds.pageRank.write.estimate(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
writeProperty: $dependencies_projection_write_property
,maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L1Norm"
})
YIELD nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView
Expand Down
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_2b_Page_Rank_Statistics.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//Centrality 2b Page Rank Statistics
// Centrality 2b Page Rank Statistics

CALL gds.pageRank.stats(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L1Norm"
})
YIELD ranIterations
Expand Down
35 changes: 35 additions & 0 deletions cypher/Centrality/Centrality_3c_Page_Rank_Mutate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// Centrality 3c Page Rank Mutate

CALL gds.pageRank.mutate(
$dependencies_projection + '-cleaned', {
maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,scaler: "L2Norm"
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,mutateProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
,didConverge
,ranIterations
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution
RETURN nodePropertiesWritten
,didConverge
,ranIterations
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution.min
,centralityDistribution.mean
,centralityDistribution.max
,centralityDistribution.p50
,centralityDistribution.p75
,centralityDistribution.p90
,centralityDistribution.p95
,centralityDistribution.p99
,centralityDistribution.p999
6 changes: 3 additions & 3 deletions ...ity/Centrality_3c_Page_Rank_Stream.cypher → ...ity/Centrality_3d_Page_Rank_Stream.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//Centrality 3c Page Rank Stream
// Centrality 3d Page Rank Stream

CALL gds.pageRank.stream(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L2Norm"
})
YIELD nodeId, score
Expand Down
6 changes: 3 additions & 3 deletions ...lity/Centrality_3d_Page_Rank_Write.cypher → ...lity/Centrality_3e_Page_Rank_Write.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//Centrality 3d Page Rank Write
// Centrality 3e Page Rank Write

CALL gds.pageRank.write(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L2Norm"
,writeProperty: $dependencies_projection_write_property
})
Expand Down
7 changes: 4 additions & 3 deletions cypher/Centrality/Centrality_4a_Article_Rank_Estimate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
//Centrality 4a Article Rank Estimate Memory
// Centrality 4a Article Rank Estimate Memory

CALL gds.articleRank.write.estimate(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
writeProperty: $dependencies_projection_write_property
,maxIterations: 30
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L1Norm"
})
YIELD nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView
Expand Down
7 changes: 4 additions & 3 deletions cypher/Centrality/Centrality_4b_Article_Rank_Statistics.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
//Centrality 4b Article Rank Statistics
// Centrality 4b Article Rank Statistics

CALL gds.articleRank.stats(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 30
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L1Norm"
})
YIELD ranIterations
Expand Down
35 changes: 35 additions & 0 deletions cypher/Centrality/Centrality_4c_Article_Rank_Mutate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
//Centrality 4c Article Rank Mutate

CALL gds.articleRank.mutate(
$dependencies_projection + '-cleaned', {
maxIterations: 30
,dampingFactor: 0.85
,tolerance: 0.00000001
,scaler: "L2Norm"
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,mutateProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
,didConverge
,ranIterations
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution
RETURN nodePropertiesWritten
,didConverge
,ranIterations
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution.min
,centralityDistribution.mean
,centralityDistribution.max
,centralityDistribution.p50
,centralityDistribution.p75
,centralityDistribution.p90
,centralityDistribution.p95
,centralityDistribution.p99
,centralityDistribution.p999
6 changes: 3 additions & 3 deletions .../Centrality_4c_Article_Rank_Stream.cypher → .../Centrality_4d_Article_Rank_Stream.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//Centrality 4c Article Rank Stream
//Centrality 4d Article Rank Stream

CALL gds.articleRank.stream(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 30
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L2Norm"
})
YIELD nodeId, score
Expand Down
6 changes: 3 additions & 3 deletions ...y/Centrality_4d_Article_Rank_Write.cypher → ...y/Centrality_4e_Article_Rank_Write.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//Centrality 4d Article Rank Write
//Centrality 4e Article Rank Write

CALL gds.articleRank.write(
$dependencies_projection + '-without-empty', {
$dependencies_projection + '-cleaned', {
maxIterations: 50
,dampingFactor: 0.85
,tolerance: 0.00000001
,relationshipWeightProperty: $dependencies_projection_weight_property
,relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,scaler: "L2Norm"
,writeProperty: $dependencies_projection_write_property
})
Expand Down
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_5a_Betweeness_Estimate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
//Centrality 5a Betweeness Estimate
// Centrality 5a Betweeness Estimate

CALL gds.betweenness.write.estimate(
$dependencies_projection + '-without-empty', {
relationshipWeightProperty: $dependencies_projection_weight_property
$dependencies_projection + '-cleaned', {
relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,writeProperty: $dependencies_projection_write_property
})
YIELD nodeCount, relationshipCount, bytesMin, bytesMax, heapPercentageMin, heapPercentageMax, treeView
Expand Down
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_5b_Betweeness_Statistics.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
//Centrality 5b Betweeness Statistics
// Centrality 5b Betweeness Statistics

CALL gds.betweenness.stats(
$dependencies_projection + '-without-empty', {
relationshipWeightProperty: $dependencies_projection_weight_property
$dependencies_projection + '-cleaned', {
relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
})
YIELD preProcessingMillis
,computeMillis
Expand Down
27 changes: 27 additions & 0 deletions cypher/Centrality/Centrality_5c_Betweeness_Mutate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Centrality 5c Betweeness Mutate

CALL gds.betweenness.mutate(
$dependencies_projection + '-cleaned', {
relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,mutateProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution
RETURN nodePropertiesWritten
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution.min
,centralityDistribution.mean
,centralityDistribution.max
,centralityDistribution.p50
,centralityDistribution.p75
,centralityDistribution.p90
,centralityDistribution.p95
,centralityDistribution.p99
,centralityDistribution.p999
6 changes: 3 additions & 3 deletions ...ty/Centrality_5c_Betweeness_Stream.cypher → ...ty/Centrality_5d_Betweeness_Stream.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
// Centrality 5c Betweeness Stream
// Centrality 5d Betweeness Stream

CALL gds.betweenness.stream(
$dependencies_projection + '-without-empty', {
relationshipWeightProperty: $dependencies_projection_weight_property
$dependencies_projection + '-cleaned', {
relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
})
YIELD nodeId, score
WITH gds.util.asNode(nodeId) AS member, score
Expand Down
6 changes: 3 additions & 3 deletions ...ity/Centrality_5d_Betweeness_Write.cypher → ...ity/Centrality_5e_Betweeness_Write.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
// Centrality 5d Betweeness Write
// Centrality 5e Betweeness Write

CALL gds.betweenness.write(
$dependencies_projection + '-without-empty', {
relationshipWeightProperty: $dependencies_projection_weight_property
$dependencies_projection + '-cleaned', {
relationshipWeightProperty: CASE $dependencies_projection_weight_property WHEN '' THEN null ELSE $dependencies_projection_weight_property END
,writeProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
Expand Down
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_6a_Cost_effective_Lazy_Forward_CELF_Estimate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Centrality 6c Cost-effective Lazy Forward (CELF) Estimate
// Centrality 6a Cost-effective Lazy Forward (CELF) Estimate

CALL gds.beta.influenceMaximization.celf.write.estimate(
$dependencies_projection + '-without-empty', {
CALL gds.influenceMaximization.celf.write.estimate(
$dependencies_projection + '-cleaned', {
seedSetSize: 5
,writeProperty: $dependencies_projection_write_property
})
Expand Down
6 changes: 3 additions & 3 deletions cypher/Centrality/Centrality_6b_Cost_effective_Lazy_Forward_CELF_Statistics.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Centrality 6b Cost-effective Lazy Forward (CELF) Estimate
// Centrality 6b Cost-effective Lazy Forward (CELF) Statistics

CALL gds.beta.influenceMaximization.celf.stats(
$dependencies_projection + '-without-empty', {
CALL gds.influenceMaximization.celf.stats(
$dependencies_projection + '-cleaned', {
seedSetSize: 5
})
YIELD computeMillis
Expand Down
17 changes: 17 additions & 0 deletions cypher/Centrality/Centrality_6c_Cost_effective_Lazy_Forward_CELF_Mutate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Centrality 6c Cost-effective Lazy Forward (CELF) Mutate

CALL gds.influenceMaximization.celf.mutate(
$dependencies_projection + '-cleaned', {
seedSetSize: 5
,mutateProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
,nodeCount
,totalSpread
,computeMillis
,mutateMillis
RETURN nodePropertiesWritten
,nodeCount
,totalSpread
,computeMillis
,mutateMillis
6 changes: 3 additions & 3 deletions ...effective_Lazy_Forward_CELF_Stream.cypher → ...effective_Lazy_Forward_CELF_Stream.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Centrality 6c Cost-effective Lazy Forward (CELF) Stream
// Centrality 6d Cost-effective Lazy Forward (CELF) Stream

CALL gds.beta.influenceMaximization.celf.stream(
$dependencies_projection + '-without-empty', {
CALL gds.influenceMaximization.celf.stream(
$dependencies_projection + '-cleaned', {
seedSetSize: 5
})
YIELD nodeId, spread
Expand Down
6 changes: 3 additions & 3 deletions ..._effective_Lazy_Forward_CELF_Write.cypher → ..._effective_Lazy_Forward_CELF_Write.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Centrality 6d Cost-effective Lazy Forward (CELF) Write
// Centrality 6e Cost-effective Lazy Forward (CELF) Write

CALL gds.beta.influenceMaximization.celf.write(
$dependencies_projection + '-without-empty', {
CALL gds.influenceMaximization.celf.write(
$dependencies_projection + '-cleaned', {
seedSetSize: 5
,writeProperty: $dependencies_projection_write_property
})
Expand Down
2 changes: 1 addition & 1 deletion cypher/Centrality/Centrality_7b_Harmonic_Closeness_Statistics.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Centrality 7b Harmonic Closeness Statistics

CALL gds.closeness.harmonic.stats(
$dependencies_projection + '-without-empty', {})
$dependencies_projection + '-cleaned', {})
YIELD preProcessingMillis
,computeMillis
,postProcessingMillis
Expand Down
26 changes: 26 additions & 0 deletions cypher/Centrality/Centrality_7c_Harmonic_Closeness_Mutate.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Centrality 7c Harmonic Closeness Mutate

CALL gds.closeness.harmonic.mutate(
$dependencies_projection + '-cleaned', {
mutateProperty: $dependencies_projection_write_property
})
YIELD nodePropertiesWritten
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution
RETURN nodePropertiesWritten
,preProcessingMillis
,computeMillis
,mutateMillis
,postProcessingMillis
,centralityDistribution.min
,centralityDistribution.mean
,centralityDistribution.max
,centralityDistribution.p50
,centralityDistribution.p75
,centralityDistribution.p90
,centralityDistribution.p95
,centralityDistribution.p99
,centralityDistribution.p999
2 changes: 1 addition & 1 deletion ...ality_7c_Harmonic_Closeness_Stream.cypher → ...ality_7d_Harmonic_Closeness_Stream.cypher
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Centrality 7a Harmonic Closeness Stream

CALL gds.alpha.closeness.harmonic.stream($dependencies_projection + '-without-empty', {})
CALL gds.closeness.harmonic.stream($dependencies_projection + '-cleaned', {})
YIELD nodeId, centrality
WITH gds.util.asNode(nodeId) AS member, centrality
RETURN coalesce(member.fqn, member.fileName, member.name) AS memberName
Expand Down
Loading