[Spark-5567][MLlib] Add predict method to LocalLDAModel

[feynmanliang]

@jkbradley @hhbyyh

Adds topicDistributions to LocalLDAModel. Please review after #7757 is merged.

[SparkQA]

Test build #38905 has finished for PR 7760 at commit 247e920.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38912 has finished for PR 7760 at commit c709dd5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38921 has finished for PR 7760 at commit 4a6f323.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hhbyyh]

use normalize?

[feynmanliang]

Thanks!

[hhbyyh]

extra line

[feynmanliang]

OK

[SparkQA]

Test build #38971 has finished for PR 7760 at commit 3be2947.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #38977 has finished for PR 7760 at commit 6bfb87c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

I just noticed this: [https://github.com//pull/7760/files#diff-965c75b823b8cbfb304a6f6774681ccaR277]
Shouldn't it scale by "count?"

[feynmanliang]

I don't think so; the bound is for the entire document (joint over all words in document), not per-word. This is needed when doing online alpha hyperparameter estimate updates. bound also is not part of the public API.

Scaling by count for perplexity is done here

[jkbradley]

We can be more specific:

Predicts the topic mixture distribution for each document (often called "theta" in the literature).  Returns a vector of zeros for an empty document.

This uses a variational approximation follow Hoffman et al. (2010), where the approximate distribution is called "gamma."  Technically, this method returns this approximation "gamma" for each document.

[feynmanliang]

OK

[jkbradley]

About scaling by counts in [https://github.com//pull/7760/files#diff-965c75b823b8cbfb304a6f6774681ccaR277]:

I don't think so; the bound is for the entire document (joint over all words in document), not per-word. This is needed when doing online alpha hyperparameter estimate updates. bound also is not part of the public API.

That bound should count each token (instance of a word), but its current implementation would treat a document "blah" and a document "blah blah" as identical.

Scaling by count for perplexity is done here

This is to make the term per-word, i.e., average over all tokens. The issue is that the value computed earlier is not quite per-word.

I think this problem might be caught by modifying the unit test to have some terms have multiple copies. (I'm trying this out currently.)

[jkbradley]

Actually, if you have gensim set up already, it might be faster for you to test it. (Thanks!)

[feynmanliang]

I will take a look tomorrow morning.

On Wed, Jul 29, 2015 at 11:40 PM jkbradley [email protected] wrote:

Actually, if you have gensim set up already, it might be faster for you to
test it. (Thanks!)

—
Reply to this email directly or view it on GitHub
#7760 (comment).

[jkbradley]

Sounds good; it's pretty darn late

[feynmanliang]

@jkbradley WRT scaling by counts:

Oh, I misunderstood you. Yes you are right, that should be scaled by counts.

I will fix it in this PR and include tests in the unit tests for logLikelihood (upcoming PR).

[SparkQA]

Test build #39075 has finished for PR 7760 at commit 0ad1134.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jkbradley]

LGTM, thanks! I'll merge this with master.