Skip to content

Update integration script #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: clif/armada_integration
Choose a base branch
from
Draft

Update integration script #39

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a808940
Update integration script
EnricoMi Feb 6, 2025
f0db7b8
Skip mage and compile via docker directly
EnricoMi Feb 7, 2025
4a11b11
Restructure workflow
EnricoMi Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 32 additions & 4 deletions .github/workflows/armada.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,37 @@ jobs:
- uses: actions/checkout@v4
with:
path: spark-armada
- uses: actions/checkout@v4
with:
repository: richscott/armada
ref: rich/scala-build-container2
path: armada
- uses: actions/checkout@v4
with:
repository: armadaproject/armada-operator
path: armada-operator
- run: |
- name: Install Scala and SBT
env:
SBT_VERSION: 1.10.7
SCALA_VERSION: 2.13.15
run: |
curl -s -O https://downloads.lightbend.com/scala/${SCALA_VERSION}/scala-${SCALA_VERSION}.deb
sudo apt install -y ./scala-${SCALA_VERSION}.deb
curl -s -L -O https://github.com/sbt/sbt/releases/download/v${SBT_VERSION}/sbt-${SBT_VERSION}.tgz
tar -xzvf ./sbt-${SBT_VERSION}.tgz
echo "$PWD/sbt/bin" >> "${GITHUB_PATH}"
- name: Build Armada Scala Client
run: |
cd armada
docker build -t armada-scala-client-builder -f build/scala-client/Dockerfile .
echo docker run --rm -u $(id -u):$(id -u) -v "$PWD:/build" -w /build armada-scala-client-builder
docker run --rm -u $(id -u):$(id -u) -v "$PWD:/build" -w /build armada-scala-client-builder
cd ..

cd armada/client/scala/scala-armada-client
sbt -Dsbt.io.implicit.relative.glob.conversion=allow publishM2
- name: Build Spark Docker image
run: |
cd spark-armada
./build/sbt package -Parmada -Pkubernetes
./bin/docker-image-tool.sh -t testing build
Expand All @@ -26,6 +52,8 @@ jobs:
make kind-all
./bin/tooling/kind load image-archive ../spark_testing.tar --name armada

- name: Submit Spark app
run: |
# sleep a bit, or we see: create queue request failed: rpc error: code = DeadlineExceeded
sleep 60

Expand All @@ -34,13 +62,13 @@ jobs:
# sleep a bit, or we see: rpc error: code = PermissionDenied desc = could not find queue "test"
sleep 60

./bin/app/armadactl submit ../spark-armada/examples/spark-driver-job.yaml
./bin/app/armadactl submit ../spark-armada/examples/spark-executor-job.yaml
./bin/app/armadactl submit ../spark-armada/examples/spark-pi-driver.yaml

# wait for the jobs to start
sleep 60

# inspect jobs
- name: Inspect k8s pods
run: |
kubectl get pods
for pod in $(kubectl get pods | grep armada | cut -d " " -f 1)
do
Expand Down
Loading