Skip to content

Commit 2cee045

Browse files
krivardkrivard
authored
Merge pull request #126 from cmu-delphi/dev/rebase-jhu-deploy
Import new Jenkins/Ansible system from jhu-deploy
2 parents 7e5e1e6 + 464fb61 commit 2cee045

20 files changed

+540
-28
lines changed

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,3 +118,8 @@ venv.bak/
118118

119119
# mypy
120120
.mypy_cache/
121+
122+
# Ansible
123+
.retry
124+
.indicators-ansible-vault-pass
125+
indicators-ansible-vault-pass

‎Jenkinsfile‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!groovy
2+
3+
// import shared library: https://github.com/cmu-delphi/jenkins-shared-library
4+
@Library('jenkins-shared-library') _
5+
6+
pipeline {
7+
8+
agent any
9+
10+
stages {
11+
12+
stage ("Environment") {
13+
when {
14+
anyOf {
15+
branch "deploy-*";
16+
changeRequest target: "deploy-*", comparator: "GLOB"
17+
}
18+
}
19+
steps {
20+
script {
21+
// Get the indicator name from the pipeline env.
22+
if ( env.CHANGE_TARGET ) {
23+
INDICATOR = env.CHANGE_TARGET.replaceAll("deploy-", "")
24+
}
25+
else if ( env.BRANCH_NAME ) {
26+
INDICATOR = env.BRANCH_NAME.replaceAll("deploy-", "")
27+
}
28+
else {
29+
INDICATOR = ""
30+
}
31+
}
32+
}
33+
}
34+
35+
stage('Build') {
36+
when {
37+
changeRequest target: "deploy-*", comparator: "GLOB"
38+
}
39+
steps {
40+
sh "jenkins/${INDICATOR}-jenkins-build.sh"
41+
}
42+
}
43+
44+
stage('Test') {
45+
when {
46+
changeRequest target: "deploy-*", comparator: "GLOB"
47+
}
48+
steps {
49+
sh "jenkins/${INDICATOR}-jenkins-test.sh"
50+
}
51+
}
52+
53+
stage('Package') {
54+
when {
55+
changeRequest target: "deploy-*", comparator: "GLOB"
56+
}
57+
steps {
58+
sh "jenkins/${INDICATOR}-jenkins-package.sh"
59+
}
60+
}
61+
62+
stage('Deploy') {
63+
when {
64+
branch "deploy-*"
65+
}
66+
steps {
67+
sh "jenkins/${INDICATOR}-jenkins-deploy.sh"
68+
}
69+
}
70+
}
71+
72+
post {
73+
always {
74+
script {
75+
/*
76+
Use slackNotifier.groovy from shared library and provide current
77+
build result as parameter.
78+
*/
79+
slackNotifier(currentBuild.currentResult)
80+
}
81+
}
82+
}
83+
}

‎README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Covidcast Indicators
2+
3+
Pipeline code and supporting libraries for the **Real-time COVID-19 Indicators** used in the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu).
4+
5+
## The indicators
6+
7+
Each subdirectory contained here that is named after an indicator has specific documentation. Please review as necessary!
8+
9+
## General workflow for indicators creation and deployment
10+
11+
**tl;dr**
12+
13+
- Create your new indicator branch from `main`.
14+
- Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files.
15+
- Make some stuff!
16+
- When your stuff works, push your `dev-*` branch to remote for review.
17+
- Consult with a platform engineer for the remaining production setup needs. They will create a branch called `deploy-*` for your indicator.
18+
- Initiate a pull request against this new branch.
19+
- If your peers like it and Jenkins approves, deploy your changes by merging the PR.
20+
- Rejoice!
21+
22+
### Starting out
23+
24+
The `main` branch should contain up-to-date code and supporting libraries. This should be your starting point when creating a new indicator.
25+
26+
```shell
27+
# Hint
28+
#
29+
git checkout main
30+
git checkout -b dev-my-feature-branch
31+
```
32+
33+
### Creating your indicator
34+
35+
Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach.
36+
37+
- Consult your peers with questions! :handshake:
38+
39+
Once you have something that runs locally and passes tests you set up your remote branch eventual review and production deployment.
40+
41+
```shell
42+
# Hint
43+
#
44+
git push -u origin dev-my-feature-branch
45+
```
46+
47+
### Setting up for review and deployment
48+
49+
Once you have your branch set up you should get in touch with a platform engineer to pair up on the remaining production needs. These include:
50+
51+
- Creating the corresponding `deploy-*` branch in the repo.
52+
- Adding the necessary Jenkins scripts for your indicator.
53+
- Preparing the runtime host with any Automation configuration necessities.
54+
- Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host.
55+
56+
Once all the last mile configuration is in place you can create a pull request against the correct `deploy-*` branch to initiate the CI/CD pipeline which will build, test, and package your indicator for deployment.
57+
58+
If everything looks ok, platform engineering has validated the last mile, and the pull request is accepted, you can merge the PR. Deployment will start automatically.
59+
60+
Hopefully it'll be a full on :tada:, after that :crossed_fingers:
61+
62+
If not, circle back and try again.
63+
64+
## Production overview
65+
66+
### Running production code
67+
68+
Currently, the production indicators all live and run on the venerable and perennially useful Delphi primary server (also known generically as "the runtime host").
69+
70+
- This is a virtual machine running RHEL 7.5 and living in CMU's Campus Cloud vSphere-based infrastructure environemnt.
71+
72+
### Delivering an indicator to the production environment
73+
74+
We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io/) and [Ansible](https://www.ansible.com/) to build, test, package, and deploy each indicator individually to the runtime host.
75+
76+
- Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are:
77+
- Environment - Sets up some environment-specific needs that the other stages depend on.
78+
- Build - Create the Python venv on the Jenkins host.
79+
- Test - Run linting and unit tests.
80+
- Package - Tar and gzip the built environment.
81+
- Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary).
82+
83+
There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release.

‎ansible/ansible-deploy.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
- hosts: runtime_host
3+
vars_files:
4+
- vars.yaml
5+
tasks:
6+
- name: Copy and unarchive the package into the indicators runtime host directory.
7+
unarchive:
8+
src: "{{ jenkins_artifact_dir }}/{{ package }}"
9+
dest: "{{ indicators_runtime_dir }}"
10+
owner: "{{ runtime_user }}"
11+
group: "{{ runtime_user }}"
12+
13+
- name: Mutate Python bin path used in venv.
14+
file:
15+
src: "{{ pyenv_python_path }}"
16+
dest: "{{ indicators_runtime_dir }}/{{ indicator }}/env/bin/python"
17+
owner: "{{ runtime_user }}"
18+
group: "{{ runtime_user }}"
19+
state: link
20+
21+
- name: Set production params file.
22+
copy:
23+
src: files/{{ indicator }}-params-prod.json
24+
dest: "{{ indicators_runtime_dir }}/{{ indicator }}/params.json"
25+
owner: "{{ runtime_user }}"
26+
group: "{{ runtime_user }}"

‎ansible/ansible.cfg‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[defaults]
2+
remote_user = indicators
3+
vault_password_file = ~/.indicators-ansible-vault-pass
4+
ansible_managed = This file is managed by Ansible.%n
5+
Template: {file}
6+
Date: %Y-%m-%d %H:%M:%S
7+
User: {uid}
8+
Host: {host}

‎ansible/files/jhu-params-prod.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"export_start_date": "2020-02-20",
3+
"static_file_dir": "./static",
4+
"export_dir": "/common/covidcast/receiving/jhu-csse/",
5+
"cache_dir": "./cache",
6+
"base_url": "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_{metric}_US.csv"
7+
}

‎ansible/inventory‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[runtime_host]
2+
delphi-master-prod-01.delphi.cmu.edu

‎ansible/vars.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
runtime_user: "indicators"
3+
jenkins_artifact_dir: "/var/lib/jenkins/artifacts"
4+
indicators_runtime_dir: "/home/{{ runtime_user }}/runtime"
5+
package: "{{ indicator }}.tar.gz" # This is passed in the Ansible invocation.
6+
python_version: "3.8.2"
7+
pyenv_python_path: "/home/{{ runtime_user }}/.pyenv/versions/{{ python_version }}/bin/python"

‎jenkins/jhu-jenkins-build.sh‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/usr/bin/env bash
2+
#
3+
# JHU: Jenkins build
4+
#
5+
6+
set -exo pipefail
7+
source ~/.bash_profile
8+
9+
#
10+
# Build
11+
#
12+
13+
local_indicator="jhu"
14+
15+
cd "${WORKSPACE}/${local_indicator}" || exit
16+
17+
# Set up venv
18+
python -m venv env
19+
source env/bin/activate
20+
pip install ../_delphi_utils_python/.
21+
pip install .

‎jenkins/jhu-jenkins-deploy.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Jenkins deploy
4+
#
5+
6+
set -exo pipefail
7+
source ~/.bash_profile
8+
9+
#
10+
# Deploy
11+
#
12+
13+
local_indicator="jhu"
14+
15+
cd "${WORKSPACE}/ansible" || exit
16+
17+
# Ansible!
18+
ansible-playbook ansible-deploy.yaml --extra-vars "indicator=${local_indicator}" -i inventory

0 commit comments

Comments
 (0)