From ee0fae00ad0294dedca6a78ed05b97ef0ddcc211 Mon Sep 17 00:00:00 2001 From: Chen Chao Date: Fri, 11 Apr 2014 14:54:58 +0800 Subject: [PATCH 1/3] update spark.default.parallelism actually, the value 8 is only valid in mesos fine-grained mode : override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8) while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers! override def defaultParallelism(): Int = { conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2)) } --- docs/configuration.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index f3bfd036f4164..c46dfbe5fabfb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -96,7 +96,11 @@ Apart from these, the following properties are also available, and may be useful Property NameDefaultMeaning spark.default.parallelism - 8 + + · Mesos fine grained mode: 8
+ · Local mode: core number of the local machine
+ · Others: total core number of all executor nodes or 2, whichever is larger + Default number of tasks to use across the cluster for distributed shuffle operations (groupByKey, reduceByKey, etc) when not set by user. From 04a979665e3be7e015fed8ab0a7bfdf7deb57b8d Mon Sep 17 00:00:00 2001 From: Chen Chao Date: Fri, 11 Apr 2014 15:38:59 +0800 Subject: [PATCH 2/3] change format --- docs/configuration.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index c46dfbe5fabfb..392bdfe6e2e99 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -97,9 +97,11 @@ Apart from these, the following properties are also available, and may be useful spark.default.parallelism - · Mesos fine grained mode: 8
- · Local mode: core number of the local machine
- · Others: total core number of all executor nodes or 2, whichever is larger + Default number of tasks to use across the cluster for distributed shuffle operations (groupByKey, From 84a7fe4f6460bd5c6c83ee3bc1f971a0c814c3b7 Mon Sep 17 00:00:00 2001 From: Chen Chao Date: Mon, 14 Apr 2014 08:52:49 +0800 Subject: [PATCH 3/3] miss at the end of every single line oh yeah, missed . fixed it. --- docs/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 392bdfe6e2e99..a3029837ff0cd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -98,9 +98,9 @@ Apart from these, the following properties are also available, and may be useful spark.default.parallelism
    -
  • Mesos fine grained mode: 8 -
  • Local mode: core number of the local machine -
  • Others: total core number of all executor nodes or 2, whichever is larger +
  • Mesos fine grained mode: 8
  • +
  • Local mode: core number of the local machine
  • +
  • Others: total core number of all executor nodes or 2, whichever is larger