Skip to content

Commit 3373fa3

Browse files
author
PJEstrada
authored
Merge pull request #12 from diffgram/rabbit-eventhandlers-support
Rabbit MQ & Eventhandlers Service
2 parents 9dfedca + d3041d9 commit 3373fa3

26 files changed

+331
-214
lines changed

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,11 @@ example.com\+5.pem
1616
example.com\+6-key.pem
1717

1818
example.com\+6.pem
19+
20+
# Chart dependencies
21+
**/charts/*.tgz
22+
ca.crt
23+
24+
ca.key
25+
26+
local-ca.crt

‎Chart.lock‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
dependencies:
2-
- name: cert-manager
3-
repository: https://charts.jetstack.io/
4-
version: v1.1.0
5-
digest: sha256:50d9686126f61b7d7b8a50112464b41ac426a483ae053b4820c9e5f953cf7b76
6-
generated: "2021-01-29T14:30:59.744116786-06:00"
2+
- name: rabbitmq
3+
repository: https://charts.bitnami.com/bitnami
4+
version: 9.1.4
5+
digest: sha256:a92c6d671ae303d36df25c5c05705ee5193e1e22a6987e1476f4f815aa9887d7
6+
generated: "2022-05-24T22:45:09.592488539-06:00"

‎Chart.yaml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,16 @@ type: application
1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
1717
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18-
version: 0.1.0
18+
version: 1.0.0
1919

2020
# This is the version number of the application being deployed. This version number should be
2121
# incremented each time you make changes to the application. Versions are not expected to
2222
# follow Semantic Versioning. They should reflect the version the application is using.
2323
# It is recommended to use it with quotes.
24-
appVersion: "0.0.1"
24+
appVersion: "0.0.1"
25+
26+
dependencies:
27+
- name: rabbitmq
28+
version: 9.1.4
29+
repository: https://charts.bitnami.com/bitnami
30+
condition: useRabbitMq

‎README.md‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -46,15 +46,41 @@ imagePullCredentials:
4646
### TLS Ceritificates
4747
#### Using minikube (For local testing)
4848
Install Cert Manager
49-
`helm repo add jetstack https://charts.jetstack.io`
49+
```
50+
helm repo add jetstack https://charts.jetstack.io
51+
helm install cert-manager --namespace default jetstack/cert-manager --set installCRDs=true
52+
```
5053

51-
`helm install cert-manager --namespace default jetstack/cert-manager --set installCRDs=true`
5254

5355
Default domain on diffgram is: `example.com` so make sure you add that to your local hosts file:
5456

5557
`echo "$(minikube ip) example.com" | sudo tee -a /etc/hosts`
5658

57-
#### Using cert-manager
59+
In order for TLS to work on your local machine, you will need to provide local certificate authorities.
60+
Otherwise your web browser will detect the certificates as invalid.
61+
62+
To do that you can generate a key and certificate like this:
63+
```
64+
# Generate key
65+
openssl genrsa -out ca.key 2048
66+
# Create CA certificate signing it with the previous key.
67+
openssl req -x509 -new -nodes -key ca.key -sha256 -subj "/CN=sampleissuer.local" -days 1024 -out ca.crt -extensions v3_ca
68+
```
69+
Now create the certificates as secrets on your minkube cluster:
70+
```angular2html
71+
kubectl create secret tls my-local-ca-key-pair --key=ca.key --cert=ca.crt
72+
```
73+
Finally Modify your `values.yaml` so that helm chart can grab the secret using cert-manager
74+
issuers. Set `tlsIssuer` to `issuer-local` and `localCaSecretName` to the name you have to the secret created above:
75+
76+
```angular2html
77+
tlsIssuer: issuer-local # One of: "issuer-local", "letsencrypt-staging", or "letsencrypt-prod"
78+
localCaSecretName: my-local-ca-key-pair
79+
80+
```
81+
82+
83+
#### Using cert-manager & Public Domains
5884

5985
1. If you want to have TLS connections, please make sure you have a domain available and access to the name servers so you can modify the records to point to the IP addresses of the ingress.
6086

@@ -69,15 +95,15 @@ Default domain on diffgram is: `example.com` so make sure you add that to your l
6995
3. Reinstall the helm chart
7096

7197

72-
`helm upgrade diffgram -f diffgram/new_updated_values_from_above_step.yaml`
98+
`helm upgrade -n diffgram-ns diffgram -f diffgram/new_updated_values_from_above_step.yaml`
7399

74100
4. After a few minutes you should be able to see the issuer and the certificate generated. You can confirm this by running:
75101
`kubectl describe issuer letsencrypt-prod`
76102

77103
## B. Installation
78104
`git clone https://github.com/diffgram/diffgram-helm/`
79105

80-
`helm install diffgram ./diffgram-helm --create-namespace`
106+
`helm install -n diffgram-ns diffgram ./diffgram-helm --create-namespace`
81107

82108
If you don't change anything on `values.yaml`. You will have the namespace `default` created on your cluster
83109

‎templates/default/configmap.yaml‎

Lines changed: 1 addition & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,29 +3,4 @@ kind: ConfigMap
33
metadata:
44
name: diffgram-default-configmap
55
data:
6-
USERDOMAIN: {{ .Values.diffgramSettings.USERDOMAIN }}
7-
DIFFGRAM_SYSTEM_MODE: {{ .Values.diffgramSettings.DIFFGRAM_SYSTEM_MODE }}
8-
DIFFGRAM_STATIC_STORAGE_PROVIDER: {{ .Values.diffgramSettings.DIFFGRAM_STATIC_STORAGE_PROVIDER }}
9-
DIFFGRAM_S3_BUCKET_NAME: {{ .Values.diffgramSettings.DIFFGRAM_S3_BUCKET_NAME }}
10-
ML__DIFFGRAM_S3_BUCKET_NAME: {{ .Values.diffgramSettings.ML__DIFFGRAM_S3_BUCKET_NAME }}
11-
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcp/sa_credentials.json # Check the volume in deployment.yaml and service_account_secret.yaml
12-
CLOUD_STORAGE_BUCKET: {{ .Values.diffgramSettings.CLOUD_STORAGE_BUCKET }}
13-
ML__CLOUD_STORAGE_BUCKET: {{ .Values.diffgramSettings.ML__CLOUD_STORAGE_BUCKET }}
14-
URL_BASE: {{ .Values.diffgramDomain }}
15-
WALRUS_SERVICE_URL_BASE: {{ .Values.diffgramSettings.WALRUS_SERVICE_URL_BASE }}
16-
SERVICE_ACCOUNT_FULL_PATH: {{ .Values.diffgramSettings.SERVICE_ACCOUNT_FULL_PATH }}
17-
DIFFGRAM_AZURE_CONTAINER_NAME: {{ .Values.diffgramSettings.DIFFGRAM_AZURE_CONTAINER_NAME }}
18-
ML__DIFFGRAM_AZURE_CONTAINER_NAME: {{ .Values.diffgramSettings.ML__DIFFGRAM_AZURE_CONTAINER_NAME }}
19-
DIFFGRAM_INSTALL_FINGERPRINT: {{ .Values.diffgramSettings.DIFFGRAM_INSTALL_FINGERPRINT }}
20-
DIFFGRAM_VERSION_TAG: {{ .Values.diffgramVersion }}
21-
DIFFGRAM_HOST_OS: {{ .Values.diffgramSettings.DIFFGRAM_HOST_OS }}
22-
DATABASE_CONNECTION_POOL_SIZE: {{ .Values.diffgramSettings.DATABASE_CONNECTION_POOL_SIZE }}
23-
PYTHONPATH: "/app:/app/shared:/"
24-
PROCESS_MEDIA_NUM_VIDEO_THREADS: {{ .Values.diffgramSettings.PROCESS_MEDIA_NUM_VIDEO_THREADS }}
25-
PROCESS_MEDIA_NUM_FRAME_THREADS: {{ .Values.diffgramSettings.PROCESS_MEDIA_NUM_FRAME_THREADS }}
26-
NEW_RELIC_LICENSE_KEY: {{ .Values.diffgramSettings.NEW_RELIC_LICENSE_KEY }}
27-
EMAIL_DOMAIN_NAME: {{ .Values.diffgramSettings.EMAIL_DOMAIN_NAME }}
28-
ALLOW_EVENTHUB: {{ .Values.diffgramSettings.ALLOW_EVENTHUB }}
29-
EMAIL_VALIDATION: {{ .Values.diffgramSettings.EMAIL_VALIDATION }}
30-
ALLOW_STRIPE_BILLING: {{ .Values.diffgramSettings.ALLOW_STRIPE_BILLING }}
31-
IS_OPEN_SOURCE: {{ .Values.diffgramSettings.IS_OPEN_SOURCE }}
6+
{{- template "diffgram.settings" . }}

‎templates/default/secrets.yaml‎

Lines changed: 1 addition & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,4 @@ metadata:
44
name: diffgram-default-secrets
55
type: Opaque
66
stringData:
7-
STRIPE_API_KEY: {{ .Values.diffgramSecrets.STRIPE_API_KEY }}
8-
DIFFGRAM_AWS_ACCESS_KEY_SECRET: {{ .Values.diffgramSecrets.DIFFGRAM_AWS_ACCESS_KEY_SECRET }}
9-
DIFFGRAM_AWS_ACCESS_KEY_ID: {{ .Values.diffgramSecrets.DIFFGRAM_AWS_ACCESS_KEY_ID }}
10-
_ANALYTICS_WRITE_KEY: {{ .Values.diffgramSecrets._ANALYTICS_WRITE_KEY }}
11-
MAILGUN_KEY: {{ .Values.diffgramSecrets.MAILGUN_KEY }}
12-
HUB_SPOT_KEY: {{ .Values.diffgramSecrets.HUB_SPOT_KEY }}
13-
SECRET_KEY: {{ .Values.diffgramSecrets.SECRET_KEY }}
14-
INTER_SERVICE_SECRET: {{ .Values.diffgramSecrets.INTER_SERVICE_SECRET }}
15-
FERNET_KEY: {{ .Values.diffgramSecrets.FERNET_KEY }}
16-
{{ if eq .Values.dbSettings.dbProvider "local"}}
17-
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@diffgram-postgres/{{ .Values.dbSettings.dbName }}"
18-
{{ end }}
19-
{{ if eq .Values.dbSettings.dbProvider "rds"}}
20-
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@postgres-rds-service/{{ .Values.dbSettings.dbName }}"
21-
{{ end }}
22-
{{ if eq .Values.dbSettings.dbProvider "azure"}}
23-
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@postgres-azure-service/{{ .Values.dbSettings.dbName }}"
24-
{{ end }}
25-
USER_PASSWORDS_SECRET: {{ .Values.diffgramSecrets.USER_PASSWORDS_SECRET }}
26-
DIFFGRAM_AZURE_CONNECTION_STRING: {{ .Values.diffgramSecrets.DIFFGRAM_AZURE_CONNECTION_STRING }}
7+
{{- template "diffgram.secrets" . }}

‎templates/diffgram_settings.tpl‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
{{- define "diffgram.settings" }}
2+
USERDOMAIN: {{ .Values.diffgramSettings.USERDOMAIN }}
3+
DIFFGRAM_SYSTEM_MODE: {{ .Values.diffgramSettings.DIFFGRAM_SYSTEM_MODE }}
4+
DIFFGRAM_STATIC_STORAGE_PROVIDER: {{ .Values.diffgramSettings.DIFFGRAM_STATIC_STORAGE_PROVIDER }}
5+
DIFFGRAM_S3_BUCKET_NAME: {{ .Values.diffgramSettings.DIFFGRAM_S3_BUCKET_NAME }}
6+
ML__DIFFGRAM_S3_BUCKET_NAME: {{ .Values.diffgramSettings.ML__DIFFGRAM_S3_BUCKET_NAME }}
7+
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcp/sa_credentials.json # Check the volume in deployment.yaml and service_account_secret.yaml
8+
CLOUD_STORAGE_BUCKET: {{ .Values.diffgramSettings.CLOUD_STORAGE_BUCKET }}
9+
ML__CLOUD_STORAGE_BUCKET: {{ .Values.diffgramSettings.ML__CLOUD_STORAGE_BUCKET }}
10+
URL_BASE: {{ .Values.diffgramDomain }}
11+
WALRUS_SERVICE_URL_BASE: https://{{ .Values.diffgramDomain }}
12+
SERVICE_ACCOUNT_FULL_PATH: {{ .Values.diffgramSettings.SERVICE_ACCOUNT_FULL_PATH }}
13+
DIFFGRAM_AZURE_CONTAINER_NAME: {{ .Values.diffgramSettings.DIFFGRAM_AZURE_CONTAINER_NAME }}
14+
ML__DIFFGRAM_AZURE_CONTAINER_NAME: {{ .Values.diffgramSettings.ML__DIFFGRAM_AZURE_CONTAINER_NAME }}
15+
DIFFGRAM_INSTALL_FINGERPRINT: {{ .Values.diffgramSettings.DIFFGRAM_INSTALL_FINGERPRINT }}
16+
DIFFGRAM_VERSION_TAG: {{ .Values.diffgramVersion }}
17+
DIFFGRAM_HOST_OS: {{ .Values.diffgramSettings.DIFFGRAM_HOST_OS }}
18+
DATABASE_CONNECTION_POOL_SIZE: {{ .Values.diffgramSettings.DATABASE_CONNECTION_POOL_SIZE }}
19+
PYTHONPATH: "/app:/app/shared:/"
20+
PROCESS_MEDIA_NUM_VIDEO_THREADS: {{ .Values.diffgramSettings.PROCESS_MEDIA_NUM_VIDEO_THREADS }}
21+
PROCESS_MEDIA_NUM_FRAME_THREADS: {{ .Values.diffgramSettings.PROCESS_MEDIA_NUM_FRAME_THREADS }}
22+
NEW_RELIC_LICENSE_KEY: {{ .Values.diffgramSettings.NEW_RELIC_LICENSE_KEY }}
23+
EMAIL_DOMAIN_NAME: {{ .Values.diffgramSettings.EMAIL_DOMAIN_NAME }}
24+
ALLOW_EVENTHUB: {{ .Values.diffgramSettings.ALLOW_EVENTHUB }}
25+
EMAIL_VALIDATION: {{ .Values.diffgramSettings.EMAIL_VALIDATION }}
26+
ALLOW_STRIPE_BILLING: {{ .Values.diffgramSettings.ALLOW_STRIPE_BILLING }}
27+
IS_OPEN_SOURCE: {{ .Values.diffgramSettings.IS_OPEN_SOURCE }}
28+
DIFFGRAM_MINIO_ENDPOINT_URL: {{.Values.diffgramSettings.DIFFGRAM_MINIO_ENDPOINT_URL}}
29+
DIFFGRAM_MINIO_ACCESS_KEY_ID: {{.Values.diffgramSettings.DIFFGRAM_MINIO_ACCESS_KEY_ID}}
30+
DIFFGRAM_MINIO_ACCESS_KEY_SECRET: {{.Values.diffgramSettings.DIFFGRAM_MINIO_ACCESS_KEY_SECRET}}
31+
DIFFGRAM_MINIO_DISABLED_SSL_VERIFY: {{.Values.diffgramSettings.DIFFGRAM_MINIO_DISABLED_SSL_VERIFY}}
32+
RABBITMQ_HOST: {{.Values.diffgramSettings.RABBITMQ_HOST}}
33+
RABBITMQ_PORT: {{.Values.diffgramSettings.RABBITMQ_PORT}}
34+
RABBITMQ_DEFAULT_USER: {{.Values.rabbitmq.auth.username}}
35+
{{- end }}

‎templates/diffgrams_secrets.tpl‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{{- define "diffgram.secrets" }}
2+
STRIPE_API_KEY: {{ .Values.diffgramSecrets.STRIPE_API_KEY }}
3+
DIFFGRAM_AWS_ACCESS_KEY_SECRET: {{ .Values.diffgramSecrets.DIFFGRAM_AWS_ACCESS_KEY_SECRET }}
4+
DIFFGRAM_AWS_ACCESS_KEY_ID: {{ .Values.diffgramSecrets.DIFFGRAM_AWS_ACCESS_KEY_ID }}
5+
_ANALYTICS_WRITE_KEY: {{ .Values.diffgramSecrets._ANALYTICS_WRITE_KEY }}
6+
MAILGUN_KEY: {{ .Values.diffgramSecrets.MAILGUN_KEY }}
7+
HUB_SPOT_KEY: {{ .Values.diffgramSecrets.HUB_SPOT_KEY }}
8+
SECRET_KEY: {{ .Values.diffgramSecrets.SECRET_KEY }}
9+
INTER_SERVICE_SECRET: {{ .Values.diffgramSecrets.INTER_SERVICE_SECRET }}
10+
FERNET_KEY: {{ .Values.diffgramSecrets.FERNET_KEY }}
11+
{{ if eq .Values.dbSettings.dbProvider "local"}}
12+
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@diffgram-postgres/{{ .Values.dbSettings.dbName }}"
13+
{{ end }}
14+
{{ if eq .Values.dbSettings.dbProvider "rds"}}
15+
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@postgres-rds-service/{{ .Values.dbSettings.dbName }}"
16+
{{ end }}
17+
{{ if eq .Values.dbSettings.dbProvider "azure"}}
18+
DATABASE_URL: "postgresql+psycopg2://{{ .Values.dbSettings.dbUser }}:{{ .Values.dbSettings.dbPassword }}@postgres-azure-service/{{ .Values.dbSettings.dbName }}"
19+
{{ end }}
20+
USER_PASSWORDS_SECRET: {{ .Values.diffgramSecrets.USER_PASSWORDS_SECRET }}
21+
DIFFGRAM_AZURE_CONNECTION_STRING: {{ .Values.diffgramSecrets.DIFFGRAM_AZURE_CONNECTION_STRING }}
22+
RABBITMQ_DEFAULT_PASS: {{ .Values.rabbitmq.auth.password }}
23+
{{- end }}

‎templates/eventhandlers/configmap.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: diffgram-eventhandlers-configmap
5+
data:
6+
{{- template "diffgram.settings" . }}

‎templates/eventhandlers/deployment.yaml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: diffgram-eventhandlers
6+
name: diffgram-eventhandlers
7+
namespace: {{ .Release.Namespace }}
8+
spec:
9+
replicas: {{ .Values.defaultService.numReplicas }}
10+
selector:
11+
matchLabels:
12+
app: diffgram-eventhandlers
13+
template:
14+
metadata:
15+
labels:
16+
app: diffgram-eventhandlers
17+
spec:
18+
{{ if .Values.nodeGroupLabel }}
19+
nodeSelector:
20+
poolName: {{ .Values.nodeGroupLabel }}
21+
{{ end }}
22+
{{ if eq .Values.diffgramEdition "enterprise"}}
23+
imagePullSecrets:
24+
- name: diffgramsecret
25+
{{ end }}
26+
volumes:
27+
{{ if eq .Values.diffgramSettings.DIFFGRAM_STATIC_STORAGE_PROVIDER "gcp"}}
28+
- name: service-account-credentials-volume
29+
secret:
30+
secretName: gcp-service-account-credentials
31+
items:
32+
- key: sa_json
33+
path: sa_credentials.json
34+
{{ end }}
35+
initContainers:
36+
- name: check-db-ready
37+
image: postgres:11
38+
{{ if eq .Values.dbSettings.dbProvider "local"}}
39+
command: ['sh', '-c',
40+
'until pg_isready -h diffgram-postgres -p 5432;
41+
do echo waiting for database; sleep 2; done;']
42+
{{ end }}
43+
{{ if eq .Values.dbSettings.dbProvider "rds"}}
44+
command: ['sh', '-c', 'until pg_isready -h postgres-rds-service -p 5432; do echo waiting for database; sleep 2; done;']
45+
{{ end }}
46+
{{ if eq .Values.dbSettings.dbProvider "azure"}}
47+
command: ['sh', '-c', 'until pg_isready -h postgres-azure-service -p 5432; do echo waiting for database; sleep 2; done;']
48+
{{ end }}
49+
containers:
50+
{{ if eq .Values.diffgramEdition "enterprise"}}
51+
- image: gcr.io/diffgram-enterprise/eventhandlers:{{ .Values.diffgramVersion }}
52+
{{ end }}
53+
{{ if eq .Values.diffgramEdition "opencore"}}
54+
- image: gcr.io/diffgram-open-core/eventhandlers:{{ .Values.diffgramVersion }}
55+
{{ end }}
56+
imagePullPolicy: Always
57+
name: diffgram-default
58+
ports:
59+
- containerPort: 8080
60+
{{ if eq .Values.diffgramSettings.DIFFGRAM_STATIC_STORAGE_PROVIDER "gcp"}}
61+
volumeMounts:
62+
- name: service-account-credentials-volume
63+
mountPath: /etc/gcp
64+
readOnly: true
65+
{{ end }}
66+
envFrom:
67+
- configMapRef:
68+
name: diffgram-eventhandlers-configmap
69+
- secretRef:
70+
name: diffgram-eventhandlers-secrets
71+
resources:
72+
requests:
73+
cpu: {{ .Values.eventHandlersService.requests.cpu }}
74+
memory: {{ .Values.eventHandlersService.requests.memory }}
75+
limits:
76+
cpu: {{ .Values.eventHandlersService.limits.cpu }}
77+
memory: {{ .Values.eventHandlersService.limits.memory }}

0 commit comments

Comments
 (0)