[SPARK-51243][CORE][ML] Configurable allow native BLAS #49986

pan3793 · 2025-02-17T09:58:38Z

What changes were proposed in this pull request?

This PR proposes introducing a new configuration spark.ml.allowNativeBlas, when setting to false, Spark always uses Java BLAS even when the native library openblas or mkl is available on the machine.

Why are the changes needed?

Currently, there are many places in the Spark codebase hardcoded to call BLAS.nativeBLAS, when NativeBLAS is available, it always uses JNIBLAS, this generally is a good idea, but I found some negative cases in our internal ML workloads, that JavaBLAS is faster than JNIBLAS, this might be caused by the native library (e.g. mkl or openblas) bugs or does not optimized for some hardware. Given that, I think we should allow users to disable NativeBLAS explicitly.

The proposed spark.ml.allowNativeBlas configuration does not strictly follow the Spark configuration system because the sparkConf is not always available on the caller side of BLAS.nativeBLAS

Does this PR introduce any user-facing change?

Add a new feature, but the default value keeps the current behavior, also update the docs to mention the added conf.

How was this patch tested?

Manual tests, will supply more context after reaching a consensus on how to expose the configuration to end users.

Was this patch authored or co-authored using generative AI tooling?

No.

pan3793 · 2025-02-17T10:04:37Z

the current approach works with spark-submit

spark-submit --conf spark.ml.allowNativeBlas=false ...

user should modify their Java command options for cases that create embedded SparkContext in the Java app

java -Dnetlib.allowNativeBlas=false ...

pan3793 · 2025-02-17T10:11:40Z

cc @zhengruifeng @panbingkun, could you please take a look? and do you have a better idea of how to implement the configuration?

zhengruifeng · 2025-02-18T01:41:15Z

I think this PR needs reviews from @srowen @WeichenXu123 and @luhenry

zhengruifeng · 2025-02-18T01:42:50Z

core/src/main/scala/org/apache/spark/SparkContext.scala

  }
+
+  private def supplementBlasOptions(conf: SparkConf): Unit = {
+    conf.getOption("spark.ml.allowNativeBlas").foreach { allowNativeBlas =>


I am not sure whether we can use a env variable like MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1

I skimmed the codebase https://github.com/luhenry/netlib and found neither sys prop nor env to disable native blas loading, so I need to introduce a new one. do you mean env var is preferred over sys prop?

I guess I mean, does this have to propagate via sys property at all versus just being used as a config in the code directly? But probably it is hard to plumb through access to the config object

@srowen I replied in #49986 (comment)

srowen · 2025-02-18T01:52:57Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+
+  private def supplementBlasOptions(conf: SparkConf): Unit = {
+    conf.getOption("spark.ml.allowNativeBlas").foreach { allowNativeBlas =>
+      def supplement(key: OptionalConfigEntry[String]): Unit = {


This is repeated so many times now - I wonder if a simple refactor is in order to have one 'supplement' function?

thanks for suggestion, will do

refactored by extracting a common method supplementJavaOpts

srowen · 2025-02-18T01:53:37Z

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

    javaOpts += s"-Djava.net.preferIPv6Addresses=${Utils.preferIPv6}"

+    sparkConf.getOption("spark.ml.allowNativeBlas").foreach { allowNativeBlas =>
+      javaOpts += s"-Dspark.ml.allowNativeBlas=$allowNativeBlas"


Do other resource managers like k8s need this? not sure

the impl almost follows how we process java.net.preferIPv6Addresses, I will do test on K8s and back here.

Do other resource managers like k8s need this? not sure

K8s does not need that change.

Spark on YARN

the code appends -Dspark.ml.allowNativeBlas=... to YARN AM process command, we should assemble a java command to let YARN RM know how to bootstrap the process

Spark on K8s

client mode, no driver Pod

cluster mode, run spark-submit(which carries all Java options from local spark-submit) in the driver Pod

so it does not need to append those Java options again.

srowen · 2025-02-18T01:54:03Z

mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala

    if (_nativeBLAS == null) {
-      _nativeBLAS =
-        try { NetlibNativeBLAS.getInstance } catch { case _: Throwable => javaBLAS }
+      _nativeBLAS = System.getProperty("spark.ml.allowNativeBlas", "true") match {


This has to be a sys property because of how early it has to be initialized?

Ideally, I think we should propagate the conf via SparkConf, and change the method signature to

- def nativeBLAS: NetlibBLAS + def nativeBLAS(allowNative: Boolean): NetlibBLAS

but I found many places call BLAS.nativeBLAS where SparkConf is unavailable, so I propose to use a sys property.

pan3793 · 2025-02-18T15:52:04Z

docs/ml-linalg-guide.md

 ```
-sudo apt-get install libopenblas-base
-sudo update-alternatives --config libblas.so.3
+sudo apt-get install libopenblas-dev


libopenblas-base is removed in Debian 12 and Ubuntu 24.04, libopenblas-dev should be used instead.

https://github.com/luhenry/netlib/blob/6835050840ead1a724e2f305875c92d7cc21f834/.github/workflows/build-and-test.yml#L31

also update-alternatives --config libblas.so.3 does not work, and it's variant in different CPU-arch OS.

root@0bef5c80cdaa:/# update-alternatives --config libblas.so.3 update-alternatives: error: no alternatives for libblas.so.3

Given it already allows using -Ddev.ludovic.netlib.lapack.nativeLib=... to choose the native libraries, I would suggest eliminating how to use alternatives to manage the OS default library in our docs.

pan3793 · 2025-02-18T15:54:11Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala

 private[spark] object BLAS extends Serializable with Logging {

-  @transient private var _javaBLAS: NetlibBLAS = _
-  @transient private var _nativeBLAS: NetlibBLAS = _


remove the duplicated instance creation and call org.apache.spark.ml.linalg.BLAS

pan3793 · 2025-02-18T15:55:29Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala

 * ARPACK routines for MLlib's vectors and matrices.
 */
-private[spark] object ARPACK extends Serializable {
+private[spark] object ARPACK extends Serializable with Logging {


should I move ARPACK and LAPACK to mllib-local, to align with BLAS?

pan3793 · 2025-02-18T16:24:55Z

@srowen I addressed your previous comments(replied inline in each thread) and found some other issues during work on this PR:

spark.executor.extraJavaOptions forbids shipping value contains -Dspark, this also affects SPARK-47383 ([SPARK-47383][CORE] Support spark.shutdown.timeout config #45504 (comment)). To fix that, we can either give an exception to spark.ml.allowNativeBlas when checking spark.executor.extraJavaOptions or choose a different prefix, e.g. netlib.allowNativeBlas
In addition to mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala, I found other three places that need to modify
- mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
- mllib/src/main/scala/org/apache/spark/mllib/linalg/ARPACK.scala
- mllib/src/main/scala/org/apache/spark/mllib/linalg/LAPACK.scala
I'm not sure if we want to have 3 configurations(e.g. spark.ml.allowNativeBlas, spark.ml.allowNativeArpack, spark.ml.allowNativeLapack) or just a single one spark.ml.allowNativeBlas(or a better name?)
the official Spark docker image does not install the BLAS native libraries, should we install them?

would be great if you could have another look, thank you in advance.

luhenry

It's been a while I haven't looked at this code, but overall it LGTM. Thanks for taking care of that!

pan3793 · 2025-04-07T02:17:40Z

@luhenry thanks for your review and approval.
@srowen @zhengruifeng would appreciate it if you could have another look. also cc @LuciferYang @yaooqinn

zhengruifeng · 2025-04-08T03:54:27Z

mllib-local/pom.xml

      <groupId>org.apache.spark</groupId>
      <artifactId>spark-tags_${scala.binary.version}</artifactId>
    </dependency>
+    <dependency>


I am still a bit worried about the dependency change.
Defer to @WeichenXu123 's review.

the dep change can be eliminated if logging is unnecessary.

pan3793 · 2025-04-21T09:05:21Z

Kindly ping @WeichenXu123

github-actions bot added ML MLLIB YARN CORE labels Feb 17, 2025

pan3793 force-pushed the SPARK-51243 branch from 265f798 to e0df38e Compare February 17, 2025 10:07

zhengruifeng reviewed Feb 18, 2025

View reviewed changes

srowen reviewed Feb 18, 2025

View reviewed changes

github-actions bot added BUILD DOCS labels Feb 18, 2025

pan3793 mentioned this pull request Feb 18, 2025

[SPARK-47383][CORE] Support spark.shutdown.timeout config #45504

Closed

pan3793 commented Feb 18, 2025

View reviewed changes

pan3793 marked this pull request as ready for review February 18, 2025 16:10

luhenry approved these changes Apr 6, 2025

View reviewed changes

zhengruifeng reviewed Apr 8, 2025

View reviewed changes

pan3793 force-pushed the SPARK-51243 branch from e3ea255 to 7d3b337 Compare July 29, 2025 05:14

pan3793 force-pushed the SPARK-51243 branch from 7d3b337 to 415443a Compare August 8, 2025 08:32

pan3793 added 2 commits September 27, 2025 17:46

[SPARK-51243][CORE][ML] Configurable allow native BLAS

e7b4de4

Remove commons/utils dependency

f1b4f65

pan3793 force-pushed the SPARK-51243 branch from 415443a to f1b4f65 Compare September 27, 2025 09:52

github-actions bot removed the BUILD label Sep 27, 2025

[SPARK-51243][CORE][ML] Configurable allow native BLAS #49986

Are you sure you want to change the base?

[SPARK-51243][CORE][ML] Configurable allow native BLAS #49986

Conversation

pan3793 commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Feb 17, 2025

Uh oh!

zhengruifeng commented Feb 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Spark on YARN

Spark on K8s

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Feb 18, 2025

Uh oh!

luhenry left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Apr 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Apr 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pan3793 commented Feb 17, 2025 •

edited

Loading

pan3793 commented Feb 17, 2025 •

edited

Loading

pan3793 Feb 18, 2025 •

edited

Loading