[SPARK-20079][yarn] Fix client AM not allocating executors after restart. #18663
Conversation
…tart. The main goal of this change is to avoid the situation described in the bug, where an AM restart in the middle of a job may cause no new executors to be allocated because of faulty logic in the reset path. The change does two things: - fixes the executor alloc manager's reset() so that it does not stop allocation after a reset() in the middle of a job - re-orders the initialization of the YarnAllocator class so that it fetches the current executor ID before triggering the reset() above. This ensures both that the new allocator gets new requests for executors, and that it starts from the correct executor id. Tested with unit tests and by manually causing AM restarts while running jobs using spark-shell in YARN mode.
|
FYI, I started by trying to clean up the referenced PR but ended up pretty much re-writing it... |
|
Test build #79692 has finished for PR 18663 at commit
|
vanzin
changed the title
| // a new one registered after the failure. This will only happen in yarn-client mode. | ||
| reset() | ||
| } | ||
| reset() |
There was a problem hiding this comment.
Is it OK to trigger reset even in the first attempt?
There was a problem hiding this comment.
Yes. There are no executors when the first attempt registers with the driver, so everything reset() does basically amounts to a no-op.
There was a problem hiding this comment.
Thanks for the explain, let me try your patch locally.
|
LGTM, I tried it locally, looks now executors can be ramped up soon after AM restart. |
|
I haven't had a chance to look at this yet, but this doesn't by chance fix the allocator re-evaluating if it needs executors all the time does it? I have seen issues where executors can idle timeout, because scheduler isn't scheduling them fast enough, might be busy or the locality wait settings interfere. it gets down to only a few executors even though it has 10000+ tasks to run still. If it doesn't I will file a separate jira for that. I'll try to review this later today |
|
That sounds like a different issue. I've seen @squito debugging issues that sound similar to that, not sure if he got down to making any scheduler changes. |
|
Test build #79940 has finished for PR 18663 at commit
|
|
retest this please |
|
Test build #80085 has finished for PR 18663 at commit
|
|
Merging to master. |
5 participants
The main goal of this change is to avoid the situation described
in the bug, where an AM restart in the middle of a job may cause
no new executors to be allocated because of faulty logic in the
reset path.
The change does two things:
stop allocation after a reset() in the middle of a job
it fetches the current executor ID before triggering the reset()
above.
This ensures both that the new allocator gets new requests for executors,
and that it starts from the correct executor id.
Tested with unit tests and by manually causing AM restarts while
running jobs using spark-shell in YARN mode.
Closes #17882