[SPARK-1405][MLLIB] LDA on GraphX

[witgo]

• Asymmetric Dirichlet priors

  Asymmetric Dirichlet priors substantially increases the robustness of LDA to variations in the number of topics and to the highly skewed word frequency distributions common in natural language. 
  

• Support 1000000 topics

   This can better cover the long tail distribution
  

• Topic de-duplication

• Add the documentation

• Add infer interface

• Add unit tests

• Add the performance test

• Optimizing the infer interface performance

• Verifying the correctness of the algorithm

The performance test:

Item	value
cluster resource	36 executors(36 cores, 216g memory)
training set	253064 document, 29696335 words, 75496 distinct words
number of iterations	150
parameter	alpha 0.01 , beta 0.01
number of topics/running time (minutes)	2000/42.26 , 10000/49.47, 100000/73.14, 1000000/125.43

conf/spark-defaults.conf:

spark.akka.frameSize   20
spark.executor.instances 36
spark.rdd.compress true
spark.executor.memory   6g
spark.default.parallelism  72
spark.broadcast.blockSize  8192
spark.storage.memoryFraction 0.2
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.spark.mllib.clustering.LDAKryoRegistrator

REFERENCES:

• Y. Wang, X. Zhao, Z. Sun, H. Yan, L. Wang, Z. Jin, L. Wang, Y. Gao, J. Zeng, Q. Yang, et al. "Peacock: Learning Long-Tail Topic Features for Industrial Applications".arXiv:1405.4402, 2014
• Hanna Wallach, David Mimno, and Andrew McCallum. "Rethinking LDA: Why priors matter". In Advances in Neural Information Processing Systems 22, pages 1973–1981, 2009.

[SparkQA]

QA tests have started for PR 2388 at commit 9860fd1.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2388 at commit 5fa02ef.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit 9860fd1.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val tokens: RDD[(TopicModeling.WordId, TopicModeling.DocId)],

[SparkQA]

QA tests have finished for PR 2388 at commit 5fa02ef.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val tokens: RDD[(TopicModeling.WordId, TopicModeling.DocId)],

[SparkQA]

QA tests have started for PR 2388 at commit dc7ef13.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit dc7ef13.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val tokens: RDD[(TopicModeling.WordId, TopicModeling.DocId)],

[SparkQA]

QA tests have started for PR 2388 at commit 3738e74.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit 3738e74.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val docs: RDD[(TopicModeling.DocId, SV)],

[SparkQA]

QA tests have started for PR 2388 at commit 0dd8ad0.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit 0dd8ad0.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val docs: RDD[(TopicModeling.DocId, SV)],

[SparkQA]

QA tests have started for PR 2388 at commit d407854.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit d407854.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val docs: RDD[(TopicModeling.DocId, SV)],

[SparkQA]

QA tests have started for PR 2388 at commit 14903b1.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit 14903b1.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val docs: RDD[(TopicModeling.DocId, SV)],

[SparkQA]

QA tests have started for PR 2388 at commit f775916.

• This patch merges cleanly.

[SparkQA]

Tests timed out after a configured wait of 120m.

[SparkQA]

QA tests have started for PR 2388 at commit bf84e7b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2388 at commit bf84e7b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class TopicModeling(@transient val docs: RDD[(TopicModeling.DocId, SSV)],

[SparkQA]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20686/

[witgo]

retest this please

[jkbradley]

@witgo Thanks for the PR! This looks like a very featureful implementation, but I think it will require some refactoring to fit in well with future development. I'll give some high-level comments for now, and can perhaps do a lower-level pass later on.

APIs

I suspect we'll have other types of topic modeling in the future, not just LDA. It would be great to think ahead for that. The simplest way is probably to rename everything as "LDA", not "topic modeling," and to minimize the public API. (Other topic models we might want later are LSA, PLSA, HDP, CTM, etc.)

This should probably go under "clustering" instead of "feature."

Code organization

Some of the code is more general than LDA and could go elsewhere in MLlib. E.g., some of the sampling methods could go in stat/ Also, minMaxIndexSearch, minMaxValueSearch, etc. (or can those be replaced using existing generic methods in Scala or Java?).

Documentation and code clarity

The current thing making this hardest to review is the lack of documentation and the difficulty in understanding what each value and method does. For documentation, it will be helpful to see comments for all classes and methods, and also inline comments explaining code where needed. For code clarity, using more descriptive variable and method names will help a lot.

Other thoughts

It would be nice to remove some experimental items (such as mergeDuplicateTopic) for now.

Thanks again!

[SparkQA]

Test build #512 has started for PR 2388 at commit fe40445.

• This patch merges cleanly.

[SparkQA]

Test build #512 has finished for PR 2388 at commit fe40445.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[debasish83]

@jkbradley we support LSA (sparse coding) and PLSA through #3221...

[jkbradley]

@debasish83 Nice, I'll take a look! @witgo Thanks for making that change.

[jkbradley]

@witgo I’m submitting a simple PR for LDA which using EM for learning. I believe that it would be good to support other learning methods such as Gibbs sampling (as in your PR), where the user can select the learning method via an LDA parameter. If you have feedback on my PR, especially the public API, please do let me know. Thanks very much!

[witgo]

Here is a sample faster branch(work in progress):
https://github.com/witgo/spark/tree/lda_MH

[mengxr]

@witgo We've merged #4047 and closed this PR. Thanks for your contribution! Please create JIRAs and propose new features that can be added to the LDA implementation in master.

[witgo]

@mengxr
I created a JIRAs SPARK-5556.