Skip to content

HADOOP-19051: Highlight Capacity Scheduler new features in release for the release 3.4.0 #6500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 26, 2024
Merged

HADOOP-19051: Highlight Capacity Scheduler new features in release for the release 3.4.0 #6500

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions hadoop-project/src/site/markdown/index.md.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,27 @@ We have enhanced the YARN Federation functionality for improved usability. The e
6. The page function of the router has been enhanced.
7. A set of commands has been added to the Router side for operating on SubClusters and Policies.

YARN Capacity Scheduler improvements
----------------------------------------

[YARN-10496](https://issues.apache.org/jira/browse/YARN-10496) Support Flexible Auto Queue Creation in Capacity Scheduler

Capacity Scheduler resource distribution mode was extended with a new allocation mode called weight mode.
Defining queue capacities with weights allows the users to use the newly added flexible queue auto creation mode.
Flexible mode now supports the dynamic creation of both **parent queues** and **leaf queues**, enabling the creation of
complex queue hierarchies application submission time.

[YARN-10888](https://issues.apache.org/jira/browse/YARN-10888) New capacity modes for Capacity Scheduler

Capacity Scheduler's resource distribution was completely refactored to be more flexible and extensible. There is a new concept
called Capacity Vectors, which allows the users to mix various resource types in the hierarchy, and also in a single queue. With
this optionally enabled feature it is now possible to define different resources with different units, like memory with GBs, vcores with
percentage values, and GPUs/FPGAs with weights, all in the same queue.

[YARN-10889](https://issues.apache.org/jira/browse/YARN-10889) Queue Creation in Capacity Scheduler - Various improvements

In addition to the two new features above, there were a number of commits for improvements and bug fixes in Capacity Scheduler.

HDFS RBF: Code Enhancements, New Features, and Bug Fixes
----------------------------------------

Expand Down