Skip to content

Commit df1c7fb

Browse files
kimoonkimkimoonkim
authored
Add Kerberos test case in CI (apache-spark-on-k8s#39)
* Add krb5-server chart * krb5-server runs * Support multiple test cases * Add cases * Support skip cleaning up * Kerberos test sets up config and keytabs * Fix bugs * Fix typo * Enable kerberos for journal nodes * Fix a bug * Fix a bug * Specify journalnode princiapls in the namenode config * Test client use kerberos * Use nn0 pod for hdfs admin commands * Run test cases back to back * Describe pod when tests fail * Debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * Fix typo * debug travis * debug travis * Do mount --make-share for travis * Works end-to-end even in Travis * Document the krb5-server chart * Minor clean up * Clean up * Use -k option
1 parent 5d1010f commit df1c7fb

File tree

25 files changed

+539
-159
lines changed

25 files changed

+539
-159
lines changed

‎.travis.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,14 @@
11
sudo: required
22

33
env:
4-
- USE_MINIKUBE_DRIVER_NONE=true USE_SUDO_MINIKUBE=true
4+
- CASES=_basic.sh
5+
- CASES=_kerberos.sh
56

67
before_script:
7-
- tests/setup.sh
8+
# Required for K8s v1.10.x. See
9+
# https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783
10+
- sudo mount --make-shared / && sudo service docker restart
11+
- USE_MINIKUBE_DRIVER_NONE=true USE_SUDO_MINIKUBE=true tests/setup.sh
812

913
script:
1014
- tests/run.sh

‎charts/hdfs-client/Chart.yaml‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,4 @@
1-
# Licensed to the Apache Software Foundation (ASF) under one or more
2-
# contributor license agreements. See the NOTICE file distributed with
3-
# this work for additional information regarding copyright ownership.
4-
# The ASF licenses this file to You under the Apache License, Version 2.0
5-
# (the "License"); you may not use this file except in compliance with
6-
# the License. You may obtain a copy of the License at
7-
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
9-
#
10-
# Unless required by applicable law or agreed to in writing, software
11-
# distributed under the License is distributed on an "AS IS" BASIS,
12-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13-
# See the License for the specific language governing permissions and
14-
# limitations under the License.
1+
apiVersion: v1
152
name: hdfs-client-k8s
16-
version: 0.2
3+
version: 0.2.0
174
description: Hadoop Distributed File System (HDFS) hosted by Kubernetes.

‎charts/hdfs-datanode-k8s/Chart.yaml‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,4 @@
1-
# Licensed to the Apache Software Foundation (ASF) under one or more
2-
# contributor license agreements. See the NOTICE file distributed with
3-
# this work for additional information regarding copyright ownership.
4-
# The ASF licenses this file to You under the Apache License, Version 2.0
5-
# (the "License"); you may not use this file except in compliance with
6-
# the License. You may obtain a copy of the License at
7-
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
9-
#
10-
# Unless required by applicable law or agreed to in writing, software
11-
# distributed under the License is distributed on an "AS IS" BASIS,
12-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13-
# See the License for the specific language governing permissions and
14-
# limitations under the License.
1+
apiVersion: v1
152
name: hdfs-datanode-k8s
16-
version: 0.2
3+
version: 0.2.0
174
description: Hadoop Distributed File System (HDFS) hosted by Kubernetes.

‎charts/hdfs-datanode-k8s/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ HDFS `datanodes` running inside a kubernetes cluster. See the other chart for
2525
If enabling Kerberos, specify necessary options. For instance,
2626
```
2727
$ helm install -n my-hdfs-datanode \
28-
--set kerberosEnabled=true,kerberosRealm=MYCOMPANY.COM hdfs-datanode-k8s
28+
--set kerberosEnabled=true \
29+
--set kerberosRealm=MYCOMPANY.COM \
30+
hdfs-datanode-k8s
2931
```
3032
The two variables above are required. For other variables, see values.yaml.
3133
If you have launched the non-HA namenode using

‎charts/hdfs-datanode-k8s/templates/datanode-daemonset.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,9 @@ spec:
4848
- name: HDFS_CONF_dfs_datanode_kerberos_principal
4949
value: hdfs/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
5050
- name: HDFS_CONF_dfs_datanode_kerberos_https_principal
51-
value: http/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
51+
value: HTTP/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
5252
- name: HDFS_CONF_dfs_web_authentication_kerberos_principal
53-
value: http/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
53+
value: HTTP/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
5454
- name: HDFS_CONF_dfs_datanode_keytab_file
5555
value: /etc/security/hdfs.keytab
5656
{{- if .Values.jsvcEnabled }}

‎charts/hdfs-journalnode-k8s/Chart.yaml‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,4 @@
1-
# Licensed to the Apache Software Foundation (ASF) under one or more
2-
# contributor license agreements. See the NOTICE file distributed with
3-
# this work for additional information regarding copyright ownership.
4-
# The ASF licenses this file to You under the Apache License, Version 2.0
5-
# (the "License"); you may not use this file except in compliance with
6-
# the License. You may obtain a copy of the License at
7-
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
9-
#
10-
# Unless required by applicable law or agreed to in writing, software
11-
# distributed under the License is distributed on an "AS IS" BASIS,
12-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13-
# See the License for the specific language governing permissions and
14-
# limitations under the License.
1+
apiVersion: v1
152
name: hdfs-journalnode-k8s
16-
version: 0.2
3+
version: 0.2.0
174
description: Hadoop Distributed File System (HDFS) hosted by Kubernetes.

‎charts/hdfs-journalnode-k8s/templates/journalnode-statefulset.yaml‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,6 @@
1313
# See the License for the specific language governing permissions and
1414
# limitations under the License.
1515

16-
# TODO: Support Kerberos for journal nodes.
17-
1816
# A headless service to create DNS records.
1917
apiVersion: v1
2018
kind: Service
@@ -78,6 +76,20 @@ spec:
7876
env:
7977
# The following env vars are listed according to low-to-high precedence order.
8078
# i.e. Whoever comes last will override the earlier value of the same variable.
79+
{{- if .Values.kerberosEnabled }}
80+
- name: CORE_CONF_hadoop_security_authentication
81+
value: kerberos
82+
- name: CORE_CONF_hadoop_security_authorization
83+
value: "true"
84+
- name: CORE_CONF_hadoop_rpc_protection
85+
value: privacy
86+
- name: HDFS_CONF_dfs_journalnode_kerberos_principal
87+
value: hdfs/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
88+
- name: HDFS_CONF_dfs_journalnode_kerberos_internal_spnego_principal
89+
value: HTTP/_HOST@{{ required "A valid kerberosRealm entry required!" .Values.kerberosRealm }}
90+
- name: HDFS_CONF_dfs_journalnode_keytab_file
91+
value: /etc/security/hdfs.keytab
92+
{{- end }}
8193
{{- range $key, $value := .Values.customHadoopConfig }}
8294
- name: {{ $key | quote }}
8395
value: {{ $value | quote }}
@@ -94,7 +106,45 @@ spec:
94106
volumeMounts:
95107
- name: editdir
96108
mountPath: /hadoop/dfs/journal
109+
{{- if .Values.kerberosEnabled }}
110+
- name: kerberos-config
111+
mountPath: /etc/krb5.conf
112+
subPath: {{ .Values.kerberosConfigFileName }}
113+
readOnly: true
114+
- name: kerberos-keytab-copy
115+
mountPath: /etc/security/
116+
readOnly: true
117+
{{- end }}
118+
{{- if .Values.kerberosEnabled }}
119+
initContainers:
120+
- name: copy-kerberos-keytab
121+
image: busybox:1.27.1
122+
command: ['sh', '-c']
123+
args:
124+
- cp /kerberos-keytabs/${MY_KERBEROS_NAME}*.keytab /kerberos-keytab-copy/hdfs.keytab
125+
env:
126+
- name: MY_KERBEROS_NAME
127+
valueFrom:
128+
fieldRef:
129+
fieldPath: metadata.name
130+
volumeMounts:
131+
- name: kerberos-keytabs
132+
mountPath: /kerberos-keytabs
133+
- name: kerberos-keytab-copy
134+
mountPath: /kerberos-keytab-copy
135+
{{- end }}
97136
restartPolicy: Always
137+
{{- if .Values.kerberosEnabled }}
138+
volumes:
139+
- name: kerberos-config
140+
configMap:
141+
name: {{ .Values.kerberosConfigMap }}
142+
- name: kerberos-keytabs
143+
secret:
144+
secretName: {{ .Values.kerberosKeytabsSecret }}
145+
- name: kerberos-keytab-copy
146+
emptyDir: {}
147+
{{- end }}
98148
{{- if .Values.podSecurityContext.enabled }}
99149
securityContext:
100150
runAsUser: {{ .Values.podSecurityContext.runAsUser }}

‎charts/hdfs-journalnode-k8s/values.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,26 @@ podSecurityContext:
2727
enabled: false
2828
runAsUser: 0
2929
fsGroup: 1000
30+
31+
# Whether or not Kerberos support is enabled.
32+
kerberosEnabled: false
33+
34+
# Required to be non-empty if Kerberos is enabled. Specify your Kerberos realm name.
35+
# This should match the realm name in your Kerberos config file.
36+
kerberosRealm: ""
37+
38+
# Effective only if Kerberos is enabled. Name of the k8s config map containing
39+
# the kerberos config file.
40+
kerberosConfigMap: kerberos-config
41+
42+
# Effective only if Kerberos is enabled. Name of the kerberos config file inside
43+
# the config map.
44+
kerberosConfigFileName: krb5.conf
45+
46+
# Effective only if Kerberos is enabled. Name of the k8s secret containing
47+
# the kerberos keytab files of per-host HDFS principals. The secret should
48+
# have multiple data items. Each data item name should be formatted as:
49+
# `HOST-NAME.keytab`
50+
# where HOST-NAME should match the cluster node
51+
# host name that each per-host hdfs principal is associated with.
52+
kerberosKeytabsSecret: hdfs-kerberos-keytabs

‎charts/hdfs-namenode-k8s/Chart.yaml‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,4 @@
1-
# Licensed to the Apache Software Foundation (ASF) under one or more
2-
# contributor license agreements. See the NOTICE file distributed with
3-
# this work for additional information regarding copyright ownership.
4-
# The ASF licenses this file to You under the Apache License, Version 2.0
5-
# (the "License"); you may not use this file except in compliance with
6-
# the License. You may obtain a copy of the License at
7-
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
9-
#
10-
# Unless required by applicable law or agreed to in writing, software
11-
# distributed under the License is distributed on an "AS IS" BASIS,
12-
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13-
# See the License for the specific language governing permissions and
14-
# limitations under the License.
1+
apiVersion: v1
152
name: hdfs-namenode-k8s
16-
version: 0.2
3+
version: 0.2.0
174
description: Hadoop Distributed File System (HDFS) hosted by Kubernetes.

‎charts/hdfs-namenode-k8s/README.md‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,11 @@ See the other chart for `datanodes`.
4848
$ kubectl create configmap kerberos-config --from-file=/etc/krb5.conf
4949
```
5050
51+
We have our own kerberos server in the `krb5-server` helm chart.
52+
Currently, this is used mainly by the integration tests. But you may
53+
choose to use this for your cluster as well. For details, see
54+
the integration test case `tests/cases/_kerberos.sh`.
55+
5156
- Generate per-host principal accounts and password keytab files for the namenode
5257
and datanode daemons. This is typically done in your Kerberos KDC host. For example,
5358
suppose the namenode will run on the k8s cluster node kube-n1.mycompany.com,
@@ -56,23 +61,23 @@ See the other chart for `datanodes`.
5661
5762
```
5863
$ kadmin.local -q "addprinc -randkey hdfs/[email protected]"
59-
$ kadmin.local -q "addprinc -randkey http/[email protected]"
64+
$ kadmin.local -q "addprinc -randkey HTTP/[email protected]"
6065
$ mkdir hdfs-keytabs
6166
$ kadmin.local -q "ktadd -norandkey \
6267
-k hdfs-keytabs/kube-n1.mycompany.com.keytab \
6368
hdfs/[email protected] \
64-
http/[email protected]"
69+
HTTP/[email protected]"
6570
6671
$ kadmin.local -q "addprinc -randkey hdfs/[email protected]"
67-
$ kadmin.local -q "addprinc -randkey http/[email protected]"
72+
$ kadmin.local -q "addprinc -randkey HTTP/[email protected]"
6873
$ kadmin.local -q "ktadd -norandkey \
6974
-k hdfs-keytabs/kube-n2.mycompany.com.keytab \
7075
hdfs/[email protected] \
71-
http/[email protected]"
76+
HTTP/[email protected]"
7277
$ kadmin.local -q "ktadd -norandkey \
7378
-k hdfs-keytabs/kube-n2.mycompany.com.keytab \
7479
hdfs/[email protected] \
75-
http/[email protected]"
80+
HTTP/[email protected]"
7681
```
7782
7883
- Create a k8s secret containing all the keytab files. This will be mounted
@@ -120,15 +125,19 @@ See the other chart for `datanodes`.
120125
If enabling Kerberos, specify necessary options. For instance,
121126
```
122127
$ helm install -n my-hdfs-namenode \
123-
--set kerberosEnabled=true,kerberosRealm=MYCOMPANY.COM hdfs-namenode-k8s
128+
--set kerberosEnabled=true \
129+
--set kerberosRealm=MYCOMPANY.COM \
130+
hdfs-namenode-k8s
124131
```
125132
The two variables above are required. For other variables, see values.yaml.
126133
127134
If also using namenode labels for Kerberos, add
128135
the namenodePinningEnabled option:
129136
```
130137
$ helm install -n my-hdfs-namenode \
131-
--set kerberosEnabled=true,kerberosRealm=MYCOMPANY.COM,namenodePinningEnabled=true \
138+
--set kerberosEnabled=true \
139+
--set kerberosRealm=MYCOMPANY.COM \
140+
--set namenodePinningEnabled=true \
132141
hdfs-namenode-k8s
133142
```
134143

0 commit comments

Comments
 (0)