[SPARK-23499][MESOS] Support for priority queues in Mesos scheduler

[pgillet]

What changes were proposed in this pull request?

I push this PR as I could not re-open the stale one #20665 .

As for Yarn or Kubernetes, Mesos users should be able to specify priority queues to define a workload management policy for queued drivers in the Mesos Cluster Dispatcher.

This would ensure scheduling order while enqueuing Spark applications for a Mesos cluster.

Why are the changes needed?

Currently, submitted drivers are kept in order of their submission: the first driver added to the queue will be the first one to be executed (FIFO), regardless of their priority.

See https://issues.apache.org/jira/projects/SPARK/issues/SPARK-23499 for more details.

Does this PR introduce any user-facing change?

The MesosClusterDispatcher UI shows now Spark jobs along with the queue to which they are submitted.

How was this patch tested?

Unit tests.
Also, this feature has been in production for 3 years now as we use a modified Spark 2.4.0 since then.

[dongjoon-hyun]

ok to test

[dongjoon-hyun]

Thank you so much for your contribution at this module, @pgillet .
I finished my first round review and I left a few comments. Could you address those please?

[SparkQA]

Test build #131110 has finished for PR 30352 at commit 29435c8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35713/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35713/

[SparkQA]

Test build #131114 has finished for PR 30352 at commit 6d9c5a9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35717/

[SparkQA]

Test build #131116 has finished for PR 30352 at commit 72d7e81.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35717/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35719/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35719/

[SparkQA]

Test build #131146 has finished for PR 30352 at commit 7df5cb6.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35750/

[SparkQA]

Test build #131153 has finished for PR 30352 at commit e5a549b.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35750/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35756/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35756/

[SparkQA]

Test build #131172 has finished for PR 30352 at commit 93250a7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you so much for your updating, @pgillet .

[dongjoon-hyun]

BTW, quick questions, @pgillet . Which Mesos versions are you using with Spark?

[pgillet]

BTW, quick questions, @pgillet . Which Mesos versions are you using with Spark?

Mesos 1.9.0

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35774/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35774/

[SparkQA]

Test build #131180 has finished for PR 30352 at commit 72e3966.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35782/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35782/

[dongjoon-hyun]

+1, LGTM. Thank you for your contribution, @pgillet .
Merged to master for Apache Spark 3.1.

[dongjoon-hyun]

You are added to the Apache Spark contributor group and SPARK-23499 is assigned to you.
Welcome to the Apache Spark community.

[pgillet]

You are added to the Apache Spark contributor group and SPARK-23499 is assigned to you.
Welcome to the Apache Spark community.

Awesome! Thank you for your time.