[SPARK-18471][MLLIB][BACKPORT-2.0] In LBFGS, avoid sending huge vectors of 0 #16279
Conversation
…rs of 0 CostFun used to send a dense vector of zeroes as a closure in a treeAggregate call. To avoid that, we replace treeAggregate by mapPartition + treeReduce, creating a zero vector inside the mapPartition block in-place. Unit test for module mllib run locally for correctness. As for performance we run an heavy optimization on our production data (50 iterations on 128 MB weight vectors) and have seen significant decrease in terms both of runtime and container being killed by lack of off-heap memory. Author: Anthony Truchet <[email protected]> Author: sethah <[email protected]>
|
I don't think we would backport this to even 2.1. You can close this. |
|
May I ask why ? There was no conflicts so no additional qualification work is required and this looks like a performance bug fix to me, not a new feature. In order to adjust our contribution policy (fix to our internal version vs pushing upstream) we would need to understand better the backporting policy: would you have any more detailed pointer at it ? |
|
The guidance such as it is is here: http://spark.apache.org/versioning-policy.html |
|
Ok, thanks for the pointer. I do agree this is a non critical judgement call . |
|
Can we close it? |
|
We have backported it to our internal version of Spark anyhow. As told above feel free to close it if you consider this is not worth officially backporting :-) |
|
Test build #3534 has finished for PR 16279 at commit
|
4 participants
Backport #16037 to 2.0 branch
What changes were proposed in this pull request?
CostFun used to send a dense vector of zeroes as a closure in a
treeAggregate call. To avoid that, we change the aggregation operations
to convert sparse vectors into dense vectors on the fly if needed and we
pass a sparse 0 vector which is lightweight.
How was this patch tested?
Unit test for module mllib run locally for correctness.
As for performance we run an heavy optimization on our production data (50 iterations on 128 MB weight vectors) and have seen significant decrease in terms both of runtime and container being killed by lack of off-heap memory.
Author: Anthony Truchet [email protected]
Author: sethah [email protected]