Skip to content

Commit 35023e8

Browse files
committed
[SPARK-42157][CORE] spark.scheduler.mode=FAIR should provide FAIR scheduler
### What changes were proposed in this pull request? Like our documentation, `spark.sheduler.mode=FAIR` should provide a `FAIR Scheduling Within an Application`. https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application ![Screenshot 2023-01-22 at 2 59 22 PM](https://user-images.githubusercontent.com/9700541/213944956-931e3a3c-d094-4455-8990-233c7966194b.png) This bug is hidden in our CI because we have `fairscheduler.xml` always as one of test resources. - https://github.com/apache/spark/blob/master/core/src/test/resources/fairscheduler.xml ### Why are the changes needed? Currently, when `spark.scheduler.mode=FAIR` is given without scheduler allocation file, Spark creates `Fair Scheduler Pools` with `FIFO` scheduler which is wrong. We need to switch the mode to `FAIR` from `FIFO`. **BEFORE** ``` $ bin/spark-shell -c spark.scheduler.mode=FAIR Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/01/22 14:47:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/01/22 14:47:38 WARN FairSchedulableBuilder: Fair Scheduler configuration file not found so jobs will be scheduled in FIFO order. To use fair scheduling, configure pools in fairscheduler.xml or set spark.scheduler.allocation.file to a file that contains the configuration. Spark context Web UI available at http://localhost:4040 ``` ![Screenshot 2023-01-22 at 2 50 38 PM](https://user-images.githubusercontent.com/9700541/213944555-6e367a33-ca58-4daf-9ba4-b0319fbe4516.png) **AFTER** ``` $ bin/spark-shell -c spark.scheduler.mode=FAIR Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/01/22 14:48:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 ``` ![Screenshot 2023-01-22 at 2 50 14 PM](https://user-images.githubusercontent.com/9700541/213944551-660aa298-638b-450c-ad61-db9e42a624b0.png) ### Does this PR introduce _any_ user-facing change? Yes, but this is a bug fix to match with Apache Spark official documentation. ### How was this patch tested? Pass the CIs. Closes apache#39703 from dongjoon-hyun/SPARK-42157. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 4d51bfa) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 1a26c7b commit 35023e8

File tree

1 file changed

+7
-4
lines changed

1 file changed

+7
-4
lines changed

core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.Path
2727

2828
import org.apache.spark.SparkContext
2929
import org.apache.spark.internal.Logging
30-
import org.apache.spark.internal.config.SCHEDULER_ALLOCATION_FILE
30+
import org.apache.spark.internal.config.{SCHEDULER_ALLOCATION_FILE, SCHEDULER_MODE}
3131
import org.apache.spark.scheduler.SchedulingMode.SchedulingMode
3232
import org.apache.spark.util.Utils
3333

@@ -86,9 +86,12 @@ private[spark] class FairSchedulableBuilder(val rootPool: Pool, sc: SparkContext
8686
logInfo(s"Creating Fair Scheduler pools from default file: $DEFAULT_SCHEDULER_FILE")
8787
Some((is, DEFAULT_SCHEDULER_FILE))
8888
} else {
89-
logWarning("Fair Scheduler configuration file not found so jobs will be scheduled in " +
90-
s"FIFO order. To use fair scheduling, configure pools in $DEFAULT_SCHEDULER_FILE or " +
91-
s"set ${SCHEDULER_ALLOCATION_FILE.key} to a file that contains the configuration.")
89+
val schedulingMode = SchedulingMode.withName(sc.conf.get(SCHEDULER_MODE))
90+
rootPool.addSchedulable(new Pool(
91+
DEFAULT_POOL_NAME, schedulingMode, DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT))
92+
logInfo("Fair scheduler configuration not found, created default pool: " +
93+
"%s, schedulingMode: %s, minShare: %d, weight: %d".format(
94+
DEFAULT_POOL_NAME, schedulingMode, DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT))
9295
None
9396
}
9497
}

0 commit comments

Comments
 (0)