[SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTimeAsSeconds #21567
Conversation
xueyumusic
changed the title
|
|
||
| private[this] val killPollingIntervalMs: Long = | ||
| conf.getTimeAsMs("spark.task.reaper.pollingInterval", "10s") | ||
| conf.getTimeAsSeconds("spark.task.reaper.pollingInterval", "10s") * 1000L |
|
Can you add tests in |
|
|
||
| /** How long to wait before killing the python worker if a task cannot be interrupted. */ | ||
| private val taskKillTimeout = env.conf.getTimeAsMs("spark.python.task.killTimeout", "2s") | ||
| private val taskKillTimeout = env.conf.getTimeAsSeconds("spark.python.task.killTimeout", "2s") * 1000L |
There was a problem hiding this comment.
Nit: Probably, we'd be better to add Ms in the suffix, => taskKillTimeoutMs
|
I have made some modification, @maropu please review the code, thanks |
|
ok to test. |
|
Test build #91959 has finished for PR 21567 at commit
|
|
cc @jiangxb1987 |
|
@xueyumusic Can you fix the error first? |
|
Test build #91964 has finished for PR 21567 at commit
|
|
retest this please |
|
Test build #91971 has finished for PR 21567 at commit
|
|
Test build #91984 has started for PR 21567 at commit |
|
retest this please |
|
Test build #92008 has finished for PR 21567 at commit
|
|
|
||
| /** How long to wait before killing the python worker if a task cannot be interrupted. */ | ||
| private val taskKillTimeout = env.conf.getTimeAsMs("spark.python.task.killTimeout", "2s") | ||
| private val taskKillTimeoutMs = env.conf.getTimeAsSeconds("spark.python.task.killTimeout", |
There was a problem hiding this comment.
The conf is not documented, but I think it's designed to accept values like 1.5s
cc @zsxwing
| // Timeout to wait for when trying to terminate a driver. | ||
| private val DRIVER_TERMINATE_TIMEOUT_MS = | ||
| conf.getTimeAsMs("spark.worker.driverTerminateTimeout", "10s") | ||
| conf.getTimeAsSeconds("spark.worker.driverTerminateTimeout", "10s").seconds.toMillis |
|
|
||
| private[this] val killPollingIntervalMs: Long = | ||
| conf.getTimeAsMs("spark.task.reaper.pollingInterval", "10s") | ||
| conf.getTimeAsSeconds("spark.task.reaper.pollingInterval", "10s").seconds.toMillis |
There was a problem hiding this comment.
Actually I believe this is change things from right to wrong. If you want to use a second value then you shall use sth like 1s.
| // Update period of progress bar, in milliseconds | ||
| private val updatePeriodMSec = | ||
| sc.getConf.getTimeAsMs("spark.ui.consoleProgress.update.interval", "200") | ||
| sc.getConf.getTimeAsMs("spark.ui.consoleProgress.update.interval", "200ms") |
There was a problem hiding this comment.
We shall document this config.
|
Overall I don't think the current logic shall be modified. However, it shall be useful to document some the configs mentioned in this PR. |
|
I see, thanks for your review and guidance, @jiangxb1987 @maropu , I will try to add related config to doc and close this PR, thank you |
5 participants
What changes were proposed in this pull request?
This PR replaces some "getTimeAsMs" with "getTimeAsSeconds". This will return a wrong value when the user specifies a value without a time unit.
How was this patch tested?
manual test
Please review http://spark.apache.org/contributing.html before opening a pull request.