YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. #4954
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
goiri
reviewed
Sep 29, 2022
| federationStateStoreService.cleanUpFinishApplicationsWithRetries(applicationId); | ||
| if (response != null) { | ||
| LOG.info("applicationId = {} remove from state store success.", | ||
| applicationId); |
There was a problem hiding this comment.
Make the var appId and single line
There was a problem hiding this comment.
Thank you very much for your help reviewing the code, I will fix it.
|
|
||
| T runWithRetries() throws Exception { | ||
| int retry = 0; | ||
| while (true) { |
There was a problem hiding this comment.
There is no such method in the Yarn Federation module, I will make a separate abstract class and put it in the yarn-server-common federation package. This method can be used by other Yarn Federation modules.
slfan1989
changed the title
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Sep 30, 2022
| public static final String FEDERATION_STATESTORE_CLEANUP_RETRY_SLEEP_TIME = | ||
| FEDERATION_PREFIX + "state-store.clean-up-retry-sleep-time"; | ||
|
|
||
| public static final long DEFAULT_FEDERATION_STATESTORE_CLEANUP_RETRY_SLEEP_TIME = 1000; |
There was a problem hiding this comment.
Thank you very much for your help reviewing the code, I will fix it.
| The default value is 1000ms. | ||
| </description> | ||
| <name>yarn.federation.state-store.clean-up-retry-sleep-time</name> | ||
| <value>1000ms</value> |
|
|
||
| public abstract class FederationActionRetry<T> { | ||
|
|
||
| public static final Logger LOG = |
...va/org/apache/hadoop/yarn/server/resourcemanager/federation/FederationStateStoreService.java
Show resolved
Hide resolved
|
|
||
| return new FederationActionRetry<Boolean>() { | ||
| @Override | ||
| public Boolean run() throws Exception { |
There was a problem hiding this comment.
Thanks for your suggestion, I will refactor this part of the code.
| stateStore = rm.getFederationStateStoreService().getStateStoreClient(); | ||
|
|
||
| // generate an [app01] and join the [SC-1] cluster. | ||
| List<ApplicationId> appIds = new ArrayList(); |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@goiri Please help to review this pr again, thank you very much! |
goiri
approved these changes
Oct 4, 2022
goiri
reviewed
Oct 5, 2022
| if (!rmApps.containsKey(applicationId)) { | ||
| try { | ||
| Boolean cleanUpSuccess = | ||
| cleanUpFinishApplicationsWithRetries(applicationId, false); |
|
|
||
| // print app cleanup log | ||
| LOG.info("cleanup finished applications size = {}, number = {} successful cleanup.", | ||
| applicationHomeSCs.size(), successCleanUpAppCount); |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@goiri Thank you very much for helping to review the code! |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
|
@slfan1989 Hi, I found that newly submitted apps may be mistakenly deleted in Delete triggered condition 1: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: YARN-11323. [Federation] Improve Router Handler FinishApps.
In YARN-8755 and YARN-7599, the function of cleaning up the Finish App has been completed, but there are some ways to implement the function that need to be optimized.
YARN-7599 calls 2 methods:
The usage method is as follows:
The usage method is as follows:
Clean up the thread to compare list 1 and list 2, and delete the apps that do not exist in list 2.
There are several problems with this approach:
For example, the cluster has 2 SubClusters, namely SubClusterA and SubClusterB; SubCluserB restarts at this time; when
the cleaning thread calls the router's interface to obtain the RM App list, it may only obtain the App list of SubClusterA.
If it is deleted at this time, some apps may be lost.
The design ideas of YARN-11323 are as follows:
Apps that are already in the completed state should be handed over to SubCluster's RM for cleanup.
Delete triggered condition 1:
We illustrate by way of example, now our Federation cluster has 2 sub-clusters, namely SubClusterA and SubClusterB
The cleaning process of SubClusterB and SubClusterA is the same.
Delete triggered condition 2: