Minor optimizations. Use safer take, tail methods. #473
Conversation
familiarize myself with the process, mostly. * got rid of SeqLike.reverse calls when sorting by descending order * replaced slice(1, length) with safer and more readable tail calls * used foldLeft when aggregating num of docs in naive bayes code
Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
|
Can one of the admins verify this patch? |
|
Jenkins, test this please. Thanks for the clean-up! Looks good to me, pending tests... |
|
Jenkins, test this please. |
|
Merged build triggered. |
|
Merged build started. |
There was a problem hiding this comment.
btw this is probably a part we can speed up and reduce gc by not creating two arrays ...
There was a problem hiding this comment.
by two arrays, i meant one in parts, and the other for the tail, and the map. We should be able to skip 2 of them.
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@izendejas do you mind updating the pull request to address my comment? Everything else looks good. |
|
Will do later today. Thanks. On Wed, Jun 25, 2014 at 1:16 AM, Reynold Xin [email protected]
|
|
This is quite out of date, so I think we should close this issue until there is further activity. |
|
I'd be happy to resuscitate this along with a few similar Scala operation simplifications. |
|
My apologies. I had to get clearance from my current employer and then I On Thu, Jul 31, 2014 at 1:48 AM, Sean Owen [email protected] wrote:
|
Conflicts: mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala
|
PS if you need a JIRA to attach this to, it's arguably similar to https://issues.apache.org/jira/browse/SPARK-2799 |
Small merge from upstream
Update jobs naming notations
* Revert "fix ut" This reverts commit 74fced8. * Revert "[SPARK-39159][SQL] Add new Dataset API for Offset" This reverts commit b2b95a2. * Revert "[SPARK-39057][SQL] Offset could work without Limit" This reverts commit cd1c8cb. * Revert "[SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression" This reverts commit 80310f5. * update spark version
…ation problem (apache#276) * CARMEL-4773: Fix role name with upper case (apache#423) * [CARMEL-4768][CARMEL-4775] Fix NPE issue when `show grant` (apache#426) * CARMEL-4768 and CARMEL-4775: Fix NPE issue * fix * [CARMEL-4874] Role grant with admin option failing on spark 3.0 (apache#473) * [CARMEL-4874] Role grant with admin option failing on spark 3.0 * Update authorization.scala --------- Co-authored-by: yumwang <[email protected]>
…parkExecuteStatementOperation (apache#473)
5 participants
Per an email thread I initiated and after feedback and clearing my ICLA, I'm requesting some minor changes be pulled to use the more readable and safer take/tail calls over slice. Also, found an unnecessary Seq instantiation to compute the min of two values. Got rid of reverse sortBys which used negation as they weren't as readable.
Did not create a JIRA as these are very minor, but can do so.