This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Commit 38287f6
Allow spark driver find shuffle pods in specified namespace (#357)
The conf property spark.kubernetes.shuffle.namespace is used to
specify the namesapce of shuffle pods.
In normal cases, only one "shuffle daemonset" is deployed and
shared by all spark pods.
The spark driver should be able to list and watch shuffle pods
in the namespace specified by user.
Note: by default, spark driver pod doesn't have authority to
list and watch shuffle pods in another namespace. Some action
is needed to grant it the authority. For example, below ABAC
policy works.
```
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind":
"Policy", "spec": {"group": "system:serviceaccounts", "namespace":
"SHUFFLE_NAMESPACE",
"resource": "pods", "readonly": true}}
```1 parent af7297e commit 38287f6
File tree
1 file changed
+3
-1
lines changed- resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes
1 file changed
+3
-1
lines changedLines changed: 3 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
| 40 | + | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| |||
50 | 51 | | |
51 | 52 | | |
52 | 53 | | |
| 54 | + | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| |||
0 commit comments