From 16c256d8206123df9487c57c6779ad2b0e0211d0 Mon Sep 17 00:00:00 2001 From: xueyu <278006819@qq.com> Date: Sat, 23 Jun 2018 21:49:37 +0800 Subject: [PATCH 1/2] add some configs --- docs/configuration.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 6aa7878fe614d..9d1279480468f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -456,6 +456,13 @@ Apart from these, the following properties are also available, and may be useful from JVM to Python worker for every task. + + spark.python.task.killTimeout + 2s + + How long to wait before killing the python worker if a task cannot be interrupted. + + spark.sql.repl.eagerEval.enabled false @@ -568,6 +575,13 @@ Apart from these, the following properties are also available, and may be useful Python binary executable to use for PySpark in both driver and executors. + + spark.worker.driverTerminateTimeout + 10s + + Timeout to wait for when trying to terminate a driver. + + ### Shuffle Behavior @@ -798,6 +812,13 @@ Apart from these, the following properties are also available, and may be useful Buffer size to use when writing to output streams, in KiB unless otherwise specified. + + spark.ui.consoleProgress.update.interval + 200 + + Update period of progress bar, in milliseconds + + spark.ui.enabled true From b7c9704525a113866509787c3906147fcab2882e Mon Sep 17 00:00:00 2001 From: xueyu <278006819@qq.com> Date: Wed, 27 Jun 2018 10:50:05 +0800 Subject: [PATCH 2/2] modify config meaning --- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 9d1279480468f..0c1a6c9edf181 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -460,7 +460,7 @@ Apart from these, the following properties are also available, and may be useful spark.python.task.killTimeout 2s - How long to wait before killing the python worker if a task cannot be interrupted. + Timeout to wait before killing the python worker when a task cannot be interrupted