[SPARK-11432][GraphX] Personalized PageRank shouldn't use uniform initialization

[moustaki]

Changes the personalized pagerank initialization to be non-uniform.

[dbtsai]

Jenkins, add to whitelist.

[dbtsai]

Jenkins, ok to test.

[SparkQA]

Test build #44711 has finished for PR 9386 at commit d7f2edc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

      .mapVertices { (id, attr) => 
        if (personalized) {
          if (id == srcId.get) resetProb else 0.0
        } else {
          resetProb
        }
      }

[dbtsai]

With src, let's do

      .mapVertices { (id, attr) => 
          if (!(id != src && personalized)) resetProb else 0.0
      }

to avoid extra unboxing of Option

[dbtsai]

Seems that the implementation of personalized page rank doesn't follow the twitter's paper entirely when initializing the initial page rank. In the paper, only the source node should be activated. This PR addresses this issue.

+cc @dwmclary @ankurdave @rxin @jegonzal for more feedback.

Thanks.

[dwmclary]

If I recall, we specifically decided against a conditional in the BSP function at that point because the branching might causes hotspots. If that's still a concern, maybe @jegonzal can comment. Otherwise, this looks good to me -- nice catch.

[dbtsai]

I think the branching should be fine since it's just initialization.

[dwmclary]

Yes, I agree, it shouldn't add overhead.

Sent from my iPhone

On Nov 2, 2015, at 4:35 PM, DB Tsai [email protected] wrote:

I think the branching should be fine since it's just initialization.

—
Reply to this email directly or view it on GitHub.

[jegonzal]

This is actually a pretty serious error since it could lead to mass being accumulated on unreachable sub-graphs. The performance implications of the above branch should be negligible.

[moustaki]

Jenkins, ok to test.

[SparkQA]

Test build #44875 has finished for PR 9386 at commit ebf0726.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

LGTM. Thanks, merged into master.

[moustaki]

Many thanks @dbtsai

[mengxr]

I backported this patch to branch-1.5 and 1.4.