[SPARK-1892][MLLIB] Adding OWL-QN optimizer for L1 regularizations. It can also handle L2 re...

[codedeft]

Adding OWL-QN optimizer for L1 regularizations. It can also handle L2 and L1 regularizations together (balanced with alpha as in elastic nets). It extends LBFGS. It uses the OWL-QN implementation from breeze (which didn't work correctly before, but it was also fixed prior to this and committed to the latest breeze). Therefore, it requires the latest version of breeze to work correctly.

[codedeft]

jira link :

https://issues.apache.org/jira/browse/SPARK-1892

[AmplabJenkins]

Can one of the admins verify this patch?

[codedeft]

To clarify - it requires the latest breeze. The OWL-QN in breeze had bugs, which I fixed. I'm not sure if David's published an official release yet but it's in the latest snapshot.

[codedeft]

I'll try to get David to publish the latest breeze and change the project file to reference the latest breeze.

[codedeft]

Breeze has been updated to 0.8. This should now work.

[debasish83]

@codedeft make sure breeze is updated in pom.xml as well if this PR is merged in....I pulled in your code to test our internal datasets...in mllib/pom.xml it is still at 0.7..please update it to 0.8...

org.scalanlp breeze_${scala.binary.version} 0.7

[codedeft]

Done!

[gzm55]

@codedeft could you fix the conflicts when merging into master?

[mengxr]

@codedeft Could you add [SPARK-1892][MLLIB] to the title of this PR? So it shows up in the result if people search for the JIRA or [MLLIB]. Thanks!

[SparkQA]

Can one of the admins verify this patch?

[debasish83]

@codedeft I am trying to test OWLQN and compare with a C++ baseline on our data but it seems it runs only one iteration...I am setting up 10 iterations and convergenceTol as 1e-4...caller code is simple:

class LogisticRegressionWithBFGS private (
private var numIterations: Int,
private var alpha: Double,
private var regParam: Double)
extends GeneralizedLinearAlgorithm[LogisticRegressionModel] with Serializable {
private val gradient = new LogisticGradient()
override val optimizer = new OWLQN(gradient).setAlpha(alpha).setRegParam(regParam)
override protected val validators = List(DataValidators.binaryLabelValidator)
override protected def createModel(weights: Vector, intercept: Double) = {
new LogisticRegressionModel(weights, intercept)
}
}

Are there any issues with the way Breeze OWLQN is called ?

[codedeft]

@debasish83 We fixed the previously broken Breeze OWLQN in Breeze 0.8 and we know that the new Breeze OWLQN works as expected. However, this particular PR does not address a (previously) existing problem in MLLib where regularization is applied to not just weights, but also the intercept. So if you are comparing against results that do L1 regularization properly (i.e. leaving out the intercept from regularization), you'll get different results.

I think @dbtsai might be working on the intercept issue. I'm not sure if that's been merged yet.

[debasish83]

I am not that much bothered with intercept right now...Say if I force intercept to 0, this should work as expected right ?

[codedeft]

@debasish83
Yes. Or at least back when I tested it 4 months ago ;(

[dbtsai]

@debasish83 and @codedeft The weighted method for OWLQN in breeze is merged scalanlp/breeze@2570911

I will submit a PR to Spark to use newer version of breeze with this feature once @dlwh publishes to this to maven. But there is still some work in mllib side to have it working properly. I'll work on this once I'm back from vacation.

[dlwh]

breeze 0.10 is released.

On Wed, Oct 1, 2014 at 2:28 AM, DB Tsai [email protected] wrote:

@debasish83 https://github.com/debasish83 and @codedeft
https://github.com/codedeft The weighted method for OWLQN in breeze is
merged scalanlp/breeze@2570911
scalanlp/breeze@2570911

I will submit a PR to Spark to use newer version of breeze with this
feature once @dlwh https://github.com/dlwh publishes to this to maven.
But there is still some work in mllib side to have it working properly.
I'll work on this once I'm back from vacation.

—
Reply to this email directly or view it on GitHub
#840 (comment).

[srowen]

Mind closing this PR?

[dbtsai]

This will be replaced by #1518 (comment)