Skip to content

Conversation

@mpjlu
Copy link

@mpjlu mpjlu commented Apr 24, 2017

What changes were proposed in this pull request?

The blockSize of MLLIB ALS is very important for ALS performance.
In our test, when the blockSize is 128, the performance is about 4X comparing with the blockSize is 4096 (default value).
The following are our test results:
BlockSize(recommendationForAll time)
128(124s), 256(160s), 512(184s), 1024(244s), 2048(332s), 4096(488s), 8192(OOM)

The Test Environment:
3 workers: each work 10 core, each work 30G memory, each work 1 executor.
The Data: User 480,000, and Item 17,000

How was this patch tested?

The existing UT

@SparkQA
Copy link

SparkQA commented Apr 24, 2017

Test build #76096 has started for PR 17739 at commit 78e060e.

@MLnick
Copy link
Contributor

MLnick commented Apr 24, 2017

Just to confirm, the #users is 48 million, #items is 1.7 million?

@MLnick
Copy link
Contributor

MLnick commented Apr 24, 2017

Or is it 48,000 and 1,700?

@mpjlu
Copy link
Author

mpjlu commented Apr 24, 2017

users is 480,000, items is 17,000. Thanks

@MLnick
Copy link
Contributor

MLnick commented Apr 24, 2017

ok. And it is the timing for recommendProductsForUsers? Or recommendUsersForProducts?

@mpjlu
Copy link
Author

mpjlu commented Apr 24, 2017

RecommandProductsForUsers. Thanks

@MLnick
Copy link
Contributor

MLnick commented Apr 24, 2017

It's interesting to see the performance difference. I've also been looking at performance of recommend all but haven't gotten to varying the block sizes just yet.

I'm potentially in favor of exposing it as a param - but what you've got here doesn't do anything to the public API so how does that help?

@mpjlu
Copy link
Author

mpjlu commented Apr 24, 2017

Thanks @MLnick . Could you please review my another PR for recommend all performance problem.
#17742.
Sorry, I forget user cannot call recommendForAll directly for this PR.

@SparkQA
Copy link

SparkQA commented Apr 24, 2017

Test build #76108 has finished for PR 17739 at commit b4e392e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76775 has finished for PR 17739 at commit 85f6c1d.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76781 has started for PR 17739 at commit cbd2402.

@SparkQA
Copy link

SparkQA commented May 11, 2017

Test build #76787 has finished for PR 17739 at commit ffe79b8.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 16, 2017

Test build #76971 has finished for PR 17739 at commit e0d6daa.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class Imputer @Since(\"2.2.0\") (@Since(\"2.2.0\") override val uid: String)
  • case class UnresolvedTableValuedFunction(
  • case class ParseToTimestamp(left: Expression, format: Option[Expression], child: Expression)
  • class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
  • class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser
  • class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf)

@mpjlu
Copy link
Author

mpjlu commented May 17, 2017

retest this please

@SparkQA
Copy link

SparkQA commented May 17, 2017

Test build #76998 has finished for PR 17739 at commit e0d6daa.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class Imputer @Since(\"2.2.0\") (@Since(\"2.2.0\") override val uid: String)
  • case class UnresolvedTableValuedFunction(
  • case class ParseToTimestamp(left: Expression, format: Option[Expression], child: Expression)
  • class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
  • class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser
  • class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf)

@mpjlu
Copy link
Author

mpjlu commented Jan 15, 2018

Because I don't have the environment to continue this work, I will close it. Thanks.

@mpjlu mpjlu closed this Jan 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants