-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler Unit Test coverage for different build cases #16992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Jenkins this is ok to test |
|
@erenavsarogullari there are currently a bunch of higher priority outstanding scheduler PRs, so I'm guessing it will take a while for anyone to get a chance to review this, just so you know. |
|
Test build #73367 has finished for PR 16992 at commit
|
|
Could you please bring this up-to-date? @erenavsarogullari |
|
Test build #79854 has finished for PR 16992 at commit
|
|
Hi @kayousterhout @markhamstra @squito, |
squito
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@erenavsarogullari sorry this has taken forever to review! lgtm, just a super tiny nit on wording in the docs.
docs/job-scheduling.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
super nit:
... and either putting a file named fairscheduler.xml on the classpath, or setting spark.scheduler.allocation.file ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed.
|
been a while so lets run tests again just to check: |
|
Test build #3893 has finished for PR 16992 at commit
|
|
Hi @squito, Thanks for the review this patch. It is ready to re-review / merge. |
|
Test build #81168 has finished for PR 16992 at commit
|
|
Test build #81169 has finished for PR 16992 at commit
|
|
merged to master thanks @erenavsarogullari , sorry again for the delays |
What changes were proposed in this pull request?
Fair Scheduler can be built via one of the following options:
spark.scheduler.allocation.fileproperty,fairscheduler.xmlinto classpath.These options are checked in order and fair-scheduler is built via first found option. If invalid path is found,
FileNotFoundExceptionwill be expected.This PR aims unit test coverage of these use cases and a minor documentation change has been added for second option(
fairscheduler.xmlinto classpath) to inform the users.Also, this PR was related with #16813 and has been created separately to keep patch content as isolated and to help the reviewers.
How was this patch tested?
Added new Unit Tests.