Skip to content

Commit 491e607

Browse files
AzfaarQureshishovnik
AzfaarQureshi
and
shovnik
authored
Migrate CircleCI workflows to GitHub Actions (1/3) (#3302)
* Added lint job Signed-off-by: Azfaar Qureshi <[email protected]> * Added test and build jobs Signed-off-by: Azfaar Qureshi <[email protected]> * adding README Signed-off-by: Azfaar Qureshi <[email protected]> * adding table to README and removing vestigial lines from workflow Signed-off-by: Azfaar Qureshi <[email protected]> Co-authored-by: Shovnik Bhattacharya <[email protected]>
1 parent e8d5028 commit 491e607

File tree

2 files changed

+208
-0
lines changed

2 files changed

+208
-0
lines changed

‎.github/README.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# GitHub Actions CI/CD
2+
3+
The purpose of this workflow is to run all continuous integration (CI) and continuous deployment (CD) jobs when needed while respecting their internal dependencies. The continuous integration jobs serve to ensure new code passes linting, unit tests and integration tests before reaching the master branch. The continuous deployment jobs serve to deploy the latest version of the code to cortex and the website when merged with master.
4+
5+
## Contributing
6+
7+
If you wish to add a new CI or CD job, add it to the existing current test-build-deploy workflow and make sure it does not prevent any of the other jobs from passing. If you wish to change any of the build or testing images, update it in all sections are containers are often reused. If you wish to add an entirely new workflow, create a new yml file with separate triggers and filters. In all cases, clearly document any changes made to the workflows, images and dependencies below.
8+
9+
## Test, Build and Deploy
10+
11+
test-build-deploy.yml specifies a workflow that runs all Cortex continuous integration and continuous deployment jobs. The workflow is triggered on every pull request and commit to master, however the CD jobs only run when changes are merged onto master . The workflow combines both CI and CD jobs, because the CD jobs are dependent on artifacts produced the CI jobs.
12+
13+
14+
## Specific Jobs
15+
16+
| Job | Description | Type |
17+
|------------------------|-------------------------------------------------------------------------------------------------------------------------------|------|
18+
| lint | Runs linting and ensures vendor directory, protos and generated documentation are consistent. | CI |
19+
| test | Runs units tests on Cassandra testing framework. | CI |
20+
| integration-configs-db | Integration tests for database configurations. | CI |
21+
| integration | Runs integration tests after upgrading golang, pulling necessary docker images and downloading necessary module dependencies. | CI |
22+
| build | Builds and saves an up-to-date Cortex image and website. | CI |
23+
| deploy_website | Deploys the latest version of Cortex website to gh-pages branch. Triggered within workflow. | CD |
24+
| deploy | Deploys the latest Cortex image. | CD |
25+
26+
## Job Dependency Graph
27+
28+
Internal dependencies between jobs illustrated below. Jobs run concurrently where possible but do not start until all jobs they depend on have completed successfully.
29+
30+
31+
![cortex_test-build-deploy](https://user-images.githubusercontent.com/20804975/95492784-9b7feb80-0969-11eb-9934-f44a4b1da498.png)
32+
33+
### Key Details
34+
35+
**Naming Convention**
36+
37+
Each step in a job has a clear name that encapsulates the purpose of the command. The convention we are using is each word in the name should be capitalized except articles and prepositions. This creates consistent labeling when looking at the progress of the current workflow on GitHub.
38+
39+
```yaml
40+
- name: Checkout Repo
41+
# commands
42+
- name: Get Dependencies
43+
# commands
44+
```
45+
46+
**Checkout Version**
47+
48+
Current build-image ships with an older version of Git which breaks github/actions@v2 so we are using actions/checkout@v1 for all jobs until the quay image is updated to ship with a more recent version of Git.
49+
50+
```yaml
51+
- name: Checkout Repo
52+
uses: actions/checkout@v1
53+
```
54+
55+
**Symbolic Link to Expected Workspace**
56+
57+
A significant number of commands in the Makefile are hardcoded with an assumed file structure of the CI container. To ensure paths specified in previous commands don’t break, a symlink was created from the hardcoded “expected” working directory `/go/src/github.com/cortexproject/cortex` to the actual working directory `$GITHUB_WORKSPACE`.
58+
59+
```yaml
60+
- name: Sym link expected path to github workspace
61+
run: |
62+
mkdir -p /go/src/github.com/cortexproject/cortex
63+
ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
64+
```
65+
66+
**Sharing Artifacts Between Jobs**
67+
68+
As of October 2020, GitHub Actions do not persist between different jobs in the same workflow. Each job is run on a fresh virtual environment (https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/introduction-to-github-actions#runners). As such, we need to upload and download artifacts to share data between jobs.
69+
70+
| Artifact | Stored In | Used By | Purpose of Storing Artifact |
71+
|-------------------------------|-----------|---------------------------------------------|-----------------------------|
72+
| website public | build | deploy_website | share data between jobs |
73+
| Docker Images | build | deploy, integration, integrations-config-db | share data between jobs |
74+
| Frontend Protobuf | build | | long term storage |
75+
| Caching Index Client Protobuf | build | | long term storage |
76+
| Ring Protobuf | build | | long term storage |
77+
| Rules Protobuf | build | | long term storage |
78+
79+
*Note:* Docker Images are zipped before uploading as a workaround. The images contain characters that are illegal in the upload-artifact action.
80+
```yaml
81+
- name: Compressing Images
82+
run: tar -zcvf images.tar.gz /tmp/images
83+
- name: Cache Images
84+
uses: actions/upload-artifact@v2
85+
with:
86+
name: Docker Images
87+
path: ./images.tar.gz
88+
```
89+
**Tags**
90+
91+
As of Oct 2020, GitHub [does not support](https://github.202132.xyzmunity/t/using-regex-for-filtering/16427/2) regex for tag filtering. The regex /^v[0-9]+(\.[0-9]+){2}(-.+|[^-.]*)$/ was approximated using the available GitHub [filter patterns](https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet)
92+
```yaml
93+
tags:
94+
- v[0-9]+.[0-9]+.[0-9]+**
95+
```

‎.github/workflows/test-build-deploy.yml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
name: ci
2+
on:
3+
push:
4+
branches: [master]
5+
tags:
6+
- v[0-9]+.[0-9]+.[0-9]+** # Tag filters not as strict due to different regex system on Github Actions
7+
pull_request:
8+
jobs:
9+
lint:
10+
runs-on: ubuntu-latest
11+
container:
12+
image: quay.io/cortexproject/build-image:update-golang-1.14.9-eb0c8d4d2
13+
steps:
14+
- name: Checkout Repo
15+
uses: actions/checkout@v1
16+
- name: Sym Link Expected Path to Workspace
17+
run: |
18+
mkdir -p /go/src/github.com/cortexproject/cortex
19+
ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
20+
- name: Lint
21+
run: make BUILD_IN_CONTAINER=false lint
22+
- name: Check Vendor Directory
23+
run: make BUILD_IN_CONTAINER=false mod-check
24+
- name: Check Protos
25+
run: make BUILD_IN_CONTAINER=false check-protos
26+
- name: Check Generated Documentation
27+
run: make BUILD_IN_CONTAINER=false check-doc
28+
- name: Check White Noise.
29+
run: make BUILD_IN_CONTAINER=false check-white-noise
30+
31+
test:
32+
runs-on: ubuntu-latest
33+
container:
34+
image: quay.io/cortexproject/build-image:update-golang-1.14.9-eb0c8d4d2
35+
services:
36+
cassandra:
37+
image: cassandra:3.11
38+
env:
39+
JVM_OPTS: "-Xms1024M -Xmx1024M"
40+
ports:
41+
- 9042:9042
42+
steps:
43+
- name: Checkout Repo
44+
uses: actions/checkout@v1
45+
- name: Sym Link Expected Path to Workspace
46+
run: |
47+
mkdir -p /go/src/github.com/cortexproject/cortex
48+
ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
49+
- name: Run Tests
50+
run: CASSANDRA_TEST_ADDRESSES=cassandra:9042 make BUILD_IN_CONTAINER=false test
51+
52+
build:
53+
runs-on: ubuntu-latest
54+
container:
55+
image: quay.io/cortexproject/build-image:update-golang-1.14.9-eb0c8d4d2
56+
steps:
57+
- name: Checkout Repo
58+
uses: actions/checkout@v1
59+
- name: Install Docker Client
60+
run: |
61+
set -x
62+
VER="17.03.0-ce"
63+
curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
64+
tar -xz -C /tmp -f /tmp/docker-$VER.tgz
65+
mv /tmp/docker/* /usr/bin
66+
- name: Sym Link Expected Path to Workspace
67+
run: |
68+
mkdir -p /go/src/github.com/cortexproject/cortex
69+
ln -s $GITHUB_WORKSPACE/* /go/src/github.com/cortexproject/cortex
70+
- name: Build Image
71+
run: |
72+
touch build-image/.uptodate
73+
make BUILD_IN_CONTAINER=false
74+
- name: Build Website
75+
run: |
76+
touch build-image/.uptodate
77+
make BUILD_IN_CONTAINER=false web-build
78+
- uses: actions/upload-artifact@v2
79+
with:
80+
name: website public
81+
path: website/public/
82+
- uses: actions/upload-artifact@v2
83+
with:
84+
name: Frontend Protobuf
85+
path: pkg/querier/frontend/frontend.pb.go
86+
- uses: actions/upload-artifact@v2
87+
with:
88+
name: Caching Index Client Protobuf
89+
path: pkg/chunk/storage/caching_index_client.pb.go
90+
- uses: actions/upload-artifact@v2
91+
with:
92+
name: Ring Protobuf
93+
path: pkg/ring/ring.pb.go
94+
- uses: actions/upload-artifact@v2
95+
with:
96+
name: Cortex Protobuf
97+
path: pkg/ingester/client/cortex.pb.go
98+
- uses: actions/upload-artifact@v2
99+
with:
100+
name: Rules Protobuf
101+
path: pkg/ruler/rules/rules.pb.go
102+
- name: Save Images
103+
run: |
104+
mkdir /tmp/images
105+
ln -s /tmp/images ./docker-images
106+
make BUILD_IN_CONTAINER=false save-images
107+
- name: Zip Images
108+
run: tar -zcvf images.tar.gz /tmp/images
109+
- name: Upload Images Artifact
110+
uses: actions/upload-artifact@v2
111+
with:
112+
name: Docker Images
113+
path: ./images.tar.gz

0 commit comments

Comments
 (0)