SPARK-5665 [DOCS] Update netlib-java documentation

[fommil]

I am the author of netlib-java and I found this documentation to be out of date. Some main points:

1. Breeze has not depended on jBLAS for some time
2. netlib-java provides a pure JVM implementation as the fallback (the original docs did not appear to be aware of this, claiming that gfortran was necessary)
3. The licensing issue is not just about LGPL: optimised natives have proprietary licenses. Building with the LGPL flag turned on really doesn't help you get past this.
4. I really think it's best to direct people to my detailed setup guide instead of trying to compress it into one sentence. It is different for each architecture, each OS, and for each backend.

I hope this helps to clear things up 😄

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

@fommil Generally speaking you're best placed to document this aspect. I think some of the changes need to be backed out though:

• (PS could you open a simple JIRA for this? it's not 100% trivial)
• Spark itself uses JBLAS. The sentence should be worded to not imply that Breeze does, but, we still need the note about the JBLAS dependency and libgfortran.
• You also removed a reference to the netlib-lgpl profile, but, that's the easier way for people building Spark to include this dependency. If you're just questioning the name of the profile, sure, but probably not worth changing the profile name just for it, and should still be documented.
• I tend to agree with pointing to the latest external docs is better. However this also removed a reference to OpenBLAS which is going to be relevant to the JBLAS dependency itself, no?

[fommil]

@srowen can you please create the JIRA? I will inevitably fill out all the wrong fields anyway. I'll push a change to this PR in a moment.

Re: JBLAS oh I didn't realise that. Why? Does it do anything that netlib-java/Breeze doesn't? My understanding is that it absolutely requires natives to be available (which is kind of weird for a java application and hence makes Spark a lot harder to setup/use), and packages its own builds of an unspecified version of BLAS (possibly ATLAS? which is even weirder). i.e. it doesn't use system optimised binaries (see my talk to find out why this is absolutely critical for high performance). FYI, I have Intel MKL on all my machines because it is ridiculously faster than the alternatives, and I am experimenting with GPU/APU/FPGA backends.

Re: lpgl flag. ok, I'll add this back but it does sound like the flag is incorrectly named. "natives" would perhaps have been better, but, legacy, meh.

Re: "OpenBLAS", no JBLAS packages its own (suboptimal) binaries. No runtime link to lib{blas,lapack}.so.3:

ldd mikiobraun-jblas-4ee5b65/src/main/resources/lib/static/Linux/amd64/libjblas_arch_flavor.so 
    linux-vdso.so.1 (0x00007fff4f98c000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f371fd71000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f371f9c8000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f371f78a000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f371f489000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f371f273000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f37202b0000)
~/Downloads ldd mikiobraun-jblas-4ee5b65/src/main/resources/lib/static/Linux/amd64/
libjblas_arch_flavor.so  sse3/                    
~/Downloads ldd mikiobraun-jblas-4ee5b65/src/main/resources/lib/static/Linux/amd64/sse3/libjblas.so 
    linux-vdso.so.1 (0x00007ffff6ffc000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f7d32f16000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7d32b6d000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f7d3292f000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7d3262e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7d32418000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7d33bf0000)

[fommil]

OK, I've just force pushed (or, sith as I like to call it 😉)

I also had a look at mllib's use of JBLAS, and, well... I really wish somebody would pay me to look and this and optimise it because it could be a lot easier to setup AND faster by using Breeze / netlib-java consistently (plus a general review/performance stress of the implementation of many of the algorithms here, which would also be fun for me to do)

[srowen]

I opened https://issues.apache.org/jira/browse/SPARK-5665. If you would, update the title of this PR to SPARK-5665 [DOCS] Update netlib-java documentation for example.

JBLAS is used for DoubleMatrix internally as far as I can tell. Have a look at the usages and see if you think you can propose an improvement. netlib-java also needs native code available to no? that's a different version of the same problem, involving a build profile change in the case of Spark. Still I'd rather have 1 rather than 2 libraries to deal with.

[srowen]

@fommil big talk, I like it! I don't think pay is on offer, but glory is. Why not try a small improvement to demonstrate?

[fommil]

@srowen updated title. Re: natives, absolutely not! The whole point of netlib-java is that it is a common API that is swapped at runtime for Java (never leave the JVM), Natives (self contained) or System Optimised Natives (requiring runtime lib{blas,lapack}.so.3).

I'd love to work on this, I really feel I could make a difference and I see no reason for JBLAS unless it is part of your API, but my list of projects is ever-increasing and I recently had to take on another project that I don't want because of a decision by typesafe to close down some sources that I was about to roll out to production.

[fommil]

My primary FOSS focus right now is getting ENSIME to 1.0 and after that I have a few private projects demanding my time.

[srowen]

It's not part of the API. I can't evaluate your claims but you seem pretty sure about them, but am not aware of the history why netlib-java wasn't used. I suppose these things don't change unless someone thinks they are worth changing. Will let @mengxr weigh in on the changes and ad for your presentation in the docs.

[fommil]

Glory, heh, yeah right. I think it'd just get me in more trouble with my wife: I already spend too much of my personal time writing FOSS.

In light of slick/slick#1052 I really don't feel like giving any more of my personal time to advancing the commercial gains of Typesafe without renumeration (recently I've lost two full weekends and two weeknights working around this). Perhaps @derekhenninger / @dickwall would be able to find a funder to pay for this work, which I would be happy to do in stages.

[fommil]

@srowen I can guess at the history. netlib-java used to require that users compile their own natives but, inspired by JBLAS, about 2 years ago I did a big spike and beat Maven into submission to cross-build for all major platforms (and some small ones too, like the Raspberry Pi) making it super-easy for end users. At that point, I sort of feel that JBLAS has reached the end of its shelf life but I don't know what other features it offers (mllib and graphx are not using anything that isn't already provided by netlib-java/breeze). The release process for netlib-java is still a nightmare, thanks to OS X, but that doesn't affect you at all... it is my cross to bare.

I've done a lot of work benchmarking different BLAS/LAPACK routines (and Intel provide me with a gratis license, bless their souls, to aid with this) as well as other mathematics routines. I strongly recommend you watch my ScalaX talk to get a feel for what is behind this.

I would also like to use the same maven build process to make more natively-enhanced mathematics routines available on the JVM (with a Java fallback, always), such as the random number generators that we use extensively in the financial services / investment sector. That's typically the bottleneck in option pricing (i.e. Black-Scholes) for example.

[mengxr]

@fommil Thanks for the update! As @srowen pointed out, MLlib still has some JBLAS code in the NNLS solver and data generators. We should be able to remove it in the next release to simplify our dependency.

Does netlib-java still need gfrotran on linux? Or is it only required when natives are called?

I would recommend removing the https://skillsmatter.com/skillscasts/5849-high-performance-linear-algebra-in-scala link from the user guide, which doesn't look like a "permanent" link. Instead, we can use the link to your slides (http://fommil.github.io/scalax14/#/). We trust that you will keep that link working. In the slides, you can put a link to the talk.

Sorry about the netlib-lgpl flag being inaccurate. I tried to use a name not attached to a particular library (netlib-java/jblas) and covers BLAS/LAPACK/ARPACK natives. We also need to warn users about the LGPL license. So it was called netlib-lgpl and it may not be worth changing it at this time.

[fommil]

@mengxr I'm pretty sure that talk link is supposed to be permanent. If it isn't, I'll be having words with skillsmatters! 😄 I don't have a youtube link or anything for it, they want people to sign up for it. Re: "ad" for my talk, I definitely didn't make any money out of that talk, nor will I! I lost 3 days of good contracting rate to attend ScalaX 😉

You really want me to make that change or have I changed your mind? 😄

[fommil]

@mengxr sorry, forgot to answer re: gfortran. That's needed only for natives. All is documented on the netlib-java webpage. If I get a lot of Spark traffic, I'll create a custom wiki page for them.

[fommil]

@mengxr "warn people about LGPL"! Ha! That old Apache FUD. Without being aware of the licenses in place, distributors should be more concerned about Intel / Apple / AMD / NVIDIA suing them for distributing their IPR... not worried about bundling some free software.

BTW, are you aware that JBLAS links to libgfortran at runtime, which is also LGPL?

[srowen]

@fommil I think you're misrepresenting the nature of the LGPL issue. It is materially different from the Apache License 2, and so bundling LGPL with Apache Licensed code means downstream consumers of Spark suddenly are bound by more than the restrictions of the AL2, and that would be surprising. It's a recipe for helping people accidentally violate the license of the LGPL code that would then be included, as it sure looks like the whole thing is AL2 (or work-alike licenses).

It's out of respect for others' choice of licensing terms, which put restrictions on redistribution that an Apache project does not meet, that LGPL code can't be redistributed. Spark does not redistribute libgfortran. That's fine, the LGPL allows linking. Clearly there's no aversion to using LGPL software per se here.

(Or do you think that the LGPL permits redistribution in a derivative work on the same terms as the AL2 -- is that what you mean the "FUD" is?)

People should be aware of licenses and either comply with software's terms or not use it. As a vendor, we most certainly care quite a lot about getting it right. I can't agree that we should all bundle whatever free software we want, because you think it's only commercial software we have to worry about. I hope that people out there wouldn't disregard the terms under which I license my free software.

[mengxr]

@fommil I didn't mean that you were advertising netlib-java. I'm only worried about the link being "permanent" or not. We put two contracts here: 1) skillsmatters won't change the talk link, 2) you won't change the slides link. I would rather keep only one, given the fact that the talk page doesn't provide more content than what is in your slides. If later it hosts talk videos, you can simply put that link in your slides and redirect people there. Is it okay?

For gfortran, is it true that if a user has native blas/lapack installed they must have gfortran installed already? If this is true, the current statement looks good to me.

[fommil]

@srowen I've had a big discussion with Apache about this and it went nowhere, even though IP laywers stepped up to back up my stance. I firmly believe the Apache stance on LGPL is FUD and driven entirely by politics. In any case, my point in this particular case is that I find it truly ridiculous that Apache cared more about the LGPL than the genuine threat of distributing proprietary binaries with Spark... which somebody might do inadvertently because they "just copied all the libs that were needed for it to run on my machine" after checking that the Spark licensing rules were solid.

@mengxr I'm teasing re: advertising 😛 I'll update to have only one link, but it'll be a real PITA for me to update the talk slides to point to the video (the org-mode/revealjs format doesn't allow arbitrary fields in the title slide). I should probably do it anyway.

Re: gfortran... no there is absolutely no requirement for a system BLAS/LAPACK to depend on the GNU Fortran library. This is because JBLAS uses GFORTRAN-compiled binaries. I have Intel MKL and it definitely doesn't require gfortran to operate.

[fommil]

@srowen btw, have you looked inside the JBLAS jars? You know there are LGPL binaries in there, right? Look for the 32 and 64 bit versions of libgfortran-3.dll in this file: http://search.maven.org/remotecontent?filepath=org/jblas/jblas/1.2.3/jblas-1.2.3.jar

unzip -l jblas-1.2.3.jar | grep fortran
   787470  2013-02-13 15:23   lib/static/Windows/x86/libgfortran-3.dll
   937472  2013-02-13 15:23   lib/static/Windows/amd64/libgfortran-3.dll

[srowen]

@fommil I don't agree with how you think about licenses, but it's off topic for purposes here. But it's important if you're saying the JBLAS distro contains LGPL code. Are these shims that link to libgfortran, or libgfortran itself? I had assumed it is the former, or else, why does one have to install the lib locally?

[fommil]

@srowen nope, it's the full LGPL thing. Those aren't the only LGPL DLLs either, there are a few others from GCC.

If you remove the JBLAS dependency, this "problem" goes away and our mutual concern for bad licenses is already taken care of in netlib-java since JNILoader is a runtime dependency (my native layers don't actually have any LGPL code in them, although they do potentially link to LGPL code at runtime).

[srowen]

I agree, looks like these are just for Windows, but they are separate from the jblas shims. They need to be removed since they can't be redistributed. I will make a JIRA for this. I don't know if that means it can't run on Windows but in any event it can't be left like that IMHO. Yes it's another reason to try to standardize on one framework. I don't think anyone opposes that; someone just has to investigate and do the legwork.

[fommil]

It's trivial work, just needs time. I'd actually really like to do it and get more involved in optimising spark, but I really can't justify it on top of other FOSS/private commitments 😭

I think the OS is irrelevant, because legally speaking it's still "distributing" whether it runs or not. sorry misunderstood you. Yeah, I think this means Spark will not work on Windows when you remove the LGPL DLLs. I think the install will be a lot easier for everybody if JBLAS is removed.

[srowen]

@fommil Yes. I mean that it can't be distributed at all in the Spark assembly. Spark does not literally redistribute the jblas artifacts.

[mengxr]

The current version looks good to me. I'm going to merge it into master and branch-1.3. Thanks for the discussion about JBLAS' license. I think we should remove JBLAS from the Spark dependency.

@srowen We use JBLAS in NNLS and data generators. Those should be easy to replace. But we might also use it in examples. Let's make a JIRA to track the progress.

[debasish83]

@mengxr NNLS is added to Breeze in this PR scalanlp/breeze#321 once David merges it, I believe you can clean up jblas dependencies completely...

[mengxr]

@debasish83 Thanks for the update! Please ping me when it gets merged.

[fommil]

Sweet! 😄

[tsailiming]

@mengxr

I was looking into using openBLAS and found that the doc has been changed, especially on the sentence for

BLAS/LAPACK libraries on worker nodes should be built without multithreading.

Is this no longer true since Spark 1.3.0 that I don't need to build OpenBLAS with USE_THREAD=0 ?

Can you also comment on the usage on Intel MKL?

[fommil]

It's not needed because it's all covered in the linked netlib-java instructions. I recommend you click through.

[tsailiming]

I couldn't find any specific information with regards to this change on netlib-java instructions.

I thought there was a specific reason that we need to to build OpenBLAS that do not use as many threads as cores because of how Spark work in terms of N tasks(threads) per executor, to avoid the stampede problem?

Can one of the Spark developers comment on this?

[fommil]

This is an OpenBLAS problem, it's not specific to Spark.