Commit b187764
committed
### What changes were proposed in this pull request?
This PR aims to add `Java 21`-based `SparkCluster` example.
### Why are the changes needed?
Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.
- apache/spark-docker#69
### Does this PR introduce _any_ user-facing change?
No, this is a new example.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f cluster-java21.yaml
```
```
$ kubectl get sparkcluster
NAME CURRENT STATE AGE
cluster-java21 RunningHealthy 9s
```
```
$ kubectl describe sparkcluster cluster-java21
Name: cluster-java21
Namespace: default
Labels: <none>
Annotations: <none>
API Version: spark.apache.org/v1alpha1
Kind: SparkCluster
Metadata:
Creation Timestamp: 2024-09-19T04:25:30Z
Finalizers:
sparkclusters.spark.apache.org/finalizer
Generation: 2
Resource Version: 96663
UID: 9421c957-380d-4a26-a266-379304bf83ee
Spec:
Cluster Tolerations:
Instance Config:
Init Workers: 3
Max Workers: 3
Min Workers: 3
Master Spec:
Runtime Versions:
Spark Version: 4.0.0-preview1
Spark Conf:
spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21
spark.master.rest.enabled: true
spark.master.rest.host: 0.0.0.0
spark.master.ui.title: Prod Spark Cluster (Java 21)
spark.ui.reverseProxy: true
Worker Spec:
Status:
Current Attempt Summary:
Attempt Info:
Id: 0
Current State:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
State Transition History:
0:
Current State Summary: Submitted
Last Transition Time: 2024-09-19T04:25:30.640072963Z
Message: Spark cluster has been submitted to Kubernetes Cluster.
1:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
Events: <none>
```
```
$ k get pod
NAME READY STATUS RESTARTS AGE
cluster-java21-master-0 1/1 Running 0 3m20s
cluster-java21-worker-0 1/1 Running 0 3m20s
cluster-java21-worker-1 1/1 Running 0 3m20s
cluster-java21-worker-2 1/1 Running 0 3m20s
spark-kubernetes-operator-778b9bbdc6-fqks9 1/1 Running 0 20m
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #130 from dongjoon-hyun/SPARK-49715.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 815cefe commit b187764
1 file changed
+32
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
0 commit comments