diff --git a/benchmarks/README.md b/benchmarks/README.md deleted file mode 100644 index 0f07e98d6..000000000 --- a/benchmarks/README.md +++ /dev/null @@ -1,90 +0,0 @@ -## Benchmarking - -We use K6 to benchmark Parseable. This document contains the results of our benchmarks and steps to run your own benchmarks in your environment to understand Parseable's performance characteristics. - -#### Configuration - -- Parseable version: `v0.3.0` -- Server Instance: AWS EC2 `c4.2xlarge` (8 vCPU, 15 GiB RAM). Refer further details [here](https://aws.amazon.com/ec2/instance-types/). -- Client Instance: AWS EC2 `c4.8xlarge` (36 vCPU, 60 GiB RAM). Refer further details [here](https://aws.amazon.com/ec2/instance-types/). - -#### Conclusion - -- Parseable is CPU bound. CPU was 100% with lot of memory and disk iops left. -- Since we had a single client, it needed much more CPU to saturate Parseable. It would be ideal to test with distributed clients. But we expect similar performance from Parseable. -- Parseable reached `32829.535634/s` in this setup. - -#### Detailed Outcome - -```bash -k6 run load.js --vus=700 --duration=5m - - /\ |‾‾| /‾‾/ /‾‾/ - /\ / \ | |/ / / / - / \/ \ | ( / ‾‾\ - / \ | |\ \ | (‾) | - / __________ \ |__| \__\ \_____/ .io - - execution: local - script: load.js - output: - - - scenarios: (100.00%) 1 scenario, 700 max VUs, 5m30s max duration (incl. graceful stop): - * default: 700 looping VUs for 5m0s (gracefulStop: 30s) - - - data_received..................: 1.5 GB 5.0 MB/s - data_sent......................: 8.0 GB 27 MB/s - http_req_blocked...............: avg=19.35µs min=0s med=4.78µs max=431.69ms p(90)=7.35µs p(95)=9.81µs - http_req_connecting............: avg=3.79µs min=0s med=0s max=73.48ms p(90)=0s p(95)=0s - http_req_duration..............: avg=76.17ms min=344.43µs med=65.01ms max=636.72ms p(90)=128.99ms p(95)=149.54ms - { expected_response:true }...: avg=76.17ms min=344.43µs med=65.01ms max=636.72ms p(90)=128.99ms p(95)=149.54ms - http_req_failed................: 0.00% ✓ 0 ✗ 9858220 - http_req_receiving.............: avg=541.7µs min=0s med=22.49µs max=218.44ms p(90)=164.95µs p(95)=389.52µs - http_req_sending...............: avg=90.17µs min=0s med=21.07µs max=485.95ms p(90)=40.19µs p(95)=146.16µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=75.54ms min=299.31µs med=64.81ms max=482.88ms p(90)=127.43ms p(95)=147.69ms - http_reqs......................: 9858220 32829.535634/s - iteration_duration.............: avg=426.2ms min=195.51ms med=422.99ms max=1.18s p(90)=499.06ms p(95)=522.91ms - iterations.....................: 492911 1641.476782/s - vus............................: 700 min=700 max=700 - vus_max........................: 700 min=700 max=700 - - -running (5m00.3s), 000/700 VUs, 492911 complete and 0 interrupted iterations -default ✓ [======================================] 700 VUs 5m0s -``` - -#### Grafana Dashboard - -![Grafana Dashboard](https://raw.githubusercontent.com/parseablehq/.github/main/images/benchmarks/grafana.png) - -NOTE: Benchmarks are nuanced and very much environment specific. So we recommend running benchmarks in the target environment to get an understanding of actual performance. - -### Run your own load tests with K6 - -We have created a [K6](https://k6.io) script to load test a Parseable instance. The script is available [here](https://raw.githubusercontent.com/parseablehq/quest/main/testcases/load.js). - -#### Pre-requisites - -- [K6](https://k6.io) installed. -- [Parseable](https://parseable.io) installed and running. - -#### Start the script - -Make sure to change the env vars as per your setup. Also fine tune `vu` and `duration` as per your needs. - -```sh -export P_URL="https://demo.parseable.io" # Parseable URL -export P_STREAM="test" # Parseable stream -export P_USERNAME="admin" # Parseable username -export P_PASSWORD="admin" # Parseable password -export P_SCHEMA_COUNT=20 # Number of different types of json formats to be sent to this stream -k6 run --vus=700 --duration=5m https://raw.githubusercontent.com/parseablehq/quest/main/testcases/load.js -``` - -## Elastic - -Currently Elastic public benchmarks published here: [https://www.elastic.co/blog/benchmarking-and-sizing-your-elasticsearch-cluster-for-logs-and-metrics](https://www.elastic.co/blog/benchmarking-and-sizing-your-elasticsearch-cluster-for-logs-and-metrics). - -As per this benchmark, Elastic is able to ingest 22000 events per second per node. Node specs: 8 vCPU, 32 GiB RAM. diff --git a/helm-releases/parseable-1.5.0.tgz b/helm-releases/parseable-1.5.0.tgz new file mode 100644 index 000000000..8cfa45480 Binary files /dev/null and b/helm-releases/parseable-1.5.0.tgz differ diff --git a/helm/Chart.yaml b/helm/Chart.yaml index 303159425..77174f66b 100644 --- a/helm/Chart.yaml +++ b/helm/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: parseable description: Helm chart for Parseable Server type: application -version: 1.4.1 -appVersion: "v1.4.0" +version: 1.5.0 +appVersion: "v1.5.0" maintainers: - name: Parseable Team email: hi@parseable.com diff --git a/helm/templates/ingestor-statefulset.yaml b/helm/templates/ingestor-statefulset.yaml index 3cb8df000..29038762c 100644 --- a/helm/templates/ingestor-statefulset.yaml +++ b/helm/templates/ingestor-statefulset.yaml @@ -18,6 +18,7 @@ metadata: name: {{ include "parseable.fullname" . }}-ingestor namespace: {{ .Release.Namespace }} labels: + {{- .Values.parseable.highAvailability.ingestor.labels | toYaml | nindent 4 }} {{- include "parseable.ingestorLabels" . | nindent 4 }} spec: selector: diff --git a/helm/templates/querier-statefulset.yaml b/helm/templates/querier-statefulset.yaml index 35aee302a..4e35abf95 100644 --- a/helm/templates/querier-statefulset.yaml +++ b/helm/templates/querier-statefulset.yaml @@ -18,6 +18,7 @@ metadata: name: {{ include "parseable.fullname" . }}-querier namespace: {{ .Release.Namespace }} labels: + {{- .Values.parseable.podLabels | toYaml | nindent 4 }} {{- include "parseable.querierLabels" . | nindent 4 }} spec: selector: @@ -75,9 +76,25 @@ spec: - mountPath: "/parseable/hot-tier" name: hot-tier-volume {{- end }} - volumes: - - emptyDir: {} - name: stage-volume + {{- if .Values.parseable.sidecar.enabled}} + - name: {{ .Chart.Name }}-sidecar + securityContext: + {{- toYaml .Values.parseable.securityContext | nindent 8 }} + image: {{ .Values.parseable.sidecar.image.repository }}:{{ .Values.parseable.sidecar.image.tag }} + imagePullPolicy: {{ .Values.parseable.sidecar.image.pullPolicy }} + command: {{ .Values.parseable.sidecar.command }} + args: {{ .Values.parseable.sidecar.args }} + env: + {{- range $key, $value := .Values.parseable.sidecar.env }} + - name: {{ $key }} + value: {{ tpl $value $ | quote }} + {{- end }} + ports: + - containerPort: {{ .Values.parseable.sidecar.ports }} + resources: + {{- toYaml .Values.parseable.sidecar.resources | nindent 10 }} + volumeMounts: {{ .Values.parseable.sidecar.volumeMounts | toYaml | nindent 10 }} + {{- end }} volumeClaimTemplates: {{- if .Values.parseable.persistence.querier.enabled }} - metadata: @@ -90,4 +107,7 @@ spec: requests: storage: {{ .Values.parseable.persistence.querier.size | quote }} {{- end }} + {{- if .Values.parseable.sidecar.enabled}} + {{- .Values.parseable.sidecar.volumeClaimTemplates | toYaml | nindent 4 }} + {{- end }} {{- end }} diff --git a/helm/templates/standalone-deployment.yaml b/helm/templates/standalone-deployment.yaml index a8a987950..f7e18e8ac 100644 --- a/helm/templates/standalone-deployment.yaml +++ b/helm/templates/standalone-deployment.yaml @@ -18,6 +18,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} labels: + {{- .Values.parseable.podLabels | toYaml | nindent 4 }} {{- include "parseable.labelsSelector" . | nindent 8 }} spec: {{- with .Values.parseable.imagePullSecrets }} diff --git a/helm/values.yaml b/helm/values.yaml index 28c4b3877..6d765563e 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -12,6 +12,10 @@ parseable: enabled: false ingestor: port: 8000 + podAnnotations: {} + labels: + app: parseable + component: ingestor count: 3 env: RUST_LOG: warn @@ -107,6 +111,38 @@ parseable: requests: cpu: 250m memory: 1Gi + ## works only when highAvailability is enabled + ## Set it to true if you want to deploy Parseable + ## Query node with a sidecar + sidecar: + enabled: false + image: + repository: busybox + tag: latest + pullPolicy: IfNotPresent + command: [] + args: [] + env: + RUST_LOG: warn + ports: 8000 + volumeMounts: + - name: test-volume + mountPath: /parseable/test + volumeClaimTemplates: + - metadata: + name: test-volume + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 1Gi + resources: + limits: + cpu: 500m + memory: 4Gi + requests: + cpu: 250m + memory: 1Gi securityContext: allowPrivilegeEscalation: false podAnnotations: @@ -122,6 +158,9 @@ parseable: fullnameOverride: "" nodeSelector: {} affinity: {} + podLabels: + app: parseable + component: query tolerations: [] ## Use this section to create ServiceMonitor object for ## this Parseable deployment. Read more on ServiceMonitor diff --git a/index.yaml b/index.yaml index f091d7d79..39d36c8dd 100644 --- a/index.yaml +++ b/index.yaml @@ -32,9 +32,32 @@ entries: - https://charts.parseable.com/helm-releases/operator-0.0.1.tgz version: 0.0.1 parseable: + - apiVersion: v2 + appVersion: v1.5.0 + created: "2024-08-24T14:39:26.208185037+05:30" + dependencies: + - condition: vector.enabled + name: vector + repository: https://helm.vector.dev + version: 0.20.1 + - condition: fluent-bit.enabled + name: fluent-bit + repository: https://fluent.github.io/helm-charts + version: 0.25.0 + description: Helm chart for Parseable Server + digest: ab7beedfb98466b174a8378dbffaa52b1978cd70b69b801074cd6cd1f44cf651 + maintainers: + - email: hi@parseable.com + name: Parseable Team + url: https://parseable.com + name: parseable + type: application + urls: + - https://charts.parseable.com/helm-releases/parseable-1.5.0.tgz + version: 1.5.0 - apiVersion: v2 appVersion: v1.4.0 - created: "2024-08-09T17:23:25.962656675+05:30" + created: "2024-08-24T14:39:26.202241069+05:30" dependencies: - condition: vector.enabled name: vector @@ -57,7 +80,7 @@ entries: version: 1.4.1 - apiVersion: v2 appVersion: v1.4.0 - created: "2024-08-09T17:23:25.958634178+05:30" + created: "2024-08-24T14:39:26.197214148+05:30" dependencies: - condition: vector.enabled name: vector @@ -80,7 +103,7 @@ entries: version: 1.4.0 - apiVersion: v2 appVersion: v1.3.0 - created: "2024-08-09T17:23:25.95633497+05:30" + created: "2024-08-24T14:39:26.190692764+05:30" dependencies: - condition: vector.enabled name: vector @@ -103,7 +126,7 @@ entries: version: 1.3.1 - apiVersion: v2 appVersion: v1.3.0 - created: "2024-08-09T17:23:25.954149815+05:30" + created: "2024-08-24T14:39:26.185523233+05:30" dependencies: - condition: vector.enabled name: vector @@ -126,7 +149,7 @@ entries: version: 1.3.0 - apiVersion: v2 appVersion: v1.2.0 - created: "2024-08-09T17:23:25.951298986+05:30" + created: "2024-08-24T14:39:26.178658213+05:30" dependencies: - condition: vector.enabled name: vector @@ -149,7 +172,7 @@ entries: version: 1.2.0 - apiVersion: v2 appVersion: v1.1.0 - created: "2024-08-09T17:23:25.949019291+05:30" + created: "2024-08-24T14:39:26.173782373+05:30" dependencies: - condition: vector.enabled name: vector @@ -172,7 +195,7 @@ entries: version: 1.1.0 - apiVersion: v2 appVersion: v1.0.0 - created: "2024-08-09T17:23:25.945466232+05:30" + created: "2024-08-24T14:39:26.168788375+05:30" dependencies: - condition: vector.enabled name: vector @@ -195,7 +218,7 @@ entries: version: 1.0.0 - apiVersion: v2 appVersion: v0.9.0 - created: "2024-08-09T17:23:25.941900474+05:30" + created: "2024-08-24T14:39:26.162906325+05:30" dependencies: - condition: vector.enabled name: vector @@ -218,7 +241,7 @@ entries: version: 0.9.0 - apiVersion: v2 appVersion: v0.8.1 - created: "2024-08-09T17:23:25.93971334+05:30" + created: "2024-08-24T14:39:26.158198133+05:30" dependencies: - condition: vector.enabled name: vector @@ -241,7 +264,7 @@ entries: version: 0.8.1 - apiVersion: v2 appVersion: v0.8.0 - created: "2024-08-09T17:23:25.937567013+05:30" + created: "2024-08-24T14:39:26.152479164+05:30" dependencies: - condition: vector.enabled name: vector @@ -264,7 +287,7 @@ entries: version: 0.8.0 - apiVersion: v2 appVersion: v0.7.3 - created: "2024-08-09T17:23:25.934158548+05:30" + created: "2024-08-24T14:39:26.144521226+05:30" dependencies: - condition: vector.enabled name: vector @@ -287,7 +310,7 @@ entries: version: 0.7.3 - apiVersion: v2 appVersion: v0.7.2 - created: "2024-08-09T17:23:25.930893098+05:30" + created: "2024-08-24T14:39:26.136530671+05:30" dependencies: - condition: vector.enabled name: vector @@ -310,7 +333,7 @@ entries: version: 0.7.2 - apiVersion: v2 appVersion: v0.7.1 - created: "2024-08-09T17:23:25.927440291+05:30" + created: "2024-08-24T14:39:26.129115639+05:30" dependencies: - condition: vector.enabled name: vector @@ -333,7 +356,7 @@ entries: version: 0.7.1 - apiVersion: v2 appVersion: v0.7.0 - created: "2024-08-09T17:23:25.923713541+05:30" + created: "2024-08-24T14:39:26.119665363+05:30" dependencies: - condition: vector.enabled name: vector @@ -356,7 +379,7 @@ entries: version: 0.7.0 - apiVersion: v2 appVersion: v0.6.2 - created: "2024-08-09T17:23:25.921583356+05:30" + created: "2024-08-24T14:39:26.112747709+05:30" dependencies: - condition: vector.enabled name: vector @@ -379,7 +402,7 @@ entries: version: 0.6.2 - apiVersion: v2 appVersion: v0.6.1 - created: "2024-08-09T17:23:25.919405115+05:30" + created: "2024-08-24T14:39:26.103201195+05:30" dependencies: - condition: vector.enabled name: vector @@ -402,7 +425,7 @@ entries: version: 0.6.1 - apiVersion: v2 appVersion: v0.6.0 - created: "2024-08-09T17:23:25.91596509+05:30" + created: "2024-08-24T14:39:26.095260411+05:30" dependencies: - condition: vector.enabled name: vector @@ -425,7 +448,7 @@ entries: version: 0.6.0 - apiVersion: v2 appVersion: v0.5.1 - created: "2024-08-09T17:23:25.912525904+05:30" + created: "2024-08-24T14:39:26.088859724+05:30" dependencies: - condition: vector.enabled name: vector @@ -448,7 +471,7 @@ entries: version: 0.5.1 - apiVersion: v2 appVersion: v0.5.0 - created: "2024-08-09T17:23:25.909040662+05:30" + created: "2024-08-24T14:39:26.080887853+05:30" dependencies: - condition: vector.enabled name: vector @@ -471,7 +494,7 @@ entries: version: 0.5.0 - apiVersion: v2 appVersion: v0.4.4 - created: "2024-08-09T17:23:25.905658751+05:30" + created: "2024-08-24T14:39:26.071733769+05:30" dependencies: - condition: vector.enabled name: vector @@ -494,7 +517,7 @@ entries: version: 0.4.5 - apiVersion: v2 appVersion: v0.4.3 - created: "2024-08-09T17:23:25.903439666+05:30" + created: "2024-08-24T14:39:26.064242816+05:30" dependencies: - condition: vector.enabled name: vector @@ -517,7 +540,7 @@ entries: version: 0.4.4 - apiVersion: v2 appVersion: v0.4.2 - created: "2024-08-09T17:23:25.901260956+05:30" + created: "2024-08-24T14:39:26.054951398+05:30" dependencies: - condition: vector.enabled name: vector @@ -540,7 +563,7 @@ entries: version: 0.4.3 - apiVersion: v2 appVersion: v0.4.1 - created: "2024-08-09T17:23:25.897726435+05:30" + created: "2024-08-24T14:39:26.045969165+05:30" dependencies: - condition: vector.enabled name: vector @@ -563,7 +586,7 @@ entries: version: 0.4.2 - apiVersion: v2 appVersion: v0.4.0 - created: "2024-08-09T17:23:25.895520131+05:30" + created: "2024-08-24T14:39:26.037848523+05:30" dependencies: - condition: vector.enabled name: vector @@ -586,7 +609,7 @@ entries: version: 0.4.1 - apiVersion: v2 appVersion: v0.4.0 - created: "2024-08-09T17:23:25.893348225+05:30" + created: "2024-08-24T14:39:26.029708596+05:30" dependencies: - condition: vector.enabled name: vector @@ -609,7 +632,7 @@ entries: version: 0.4.0 - apiVersion: v2 appVersion: v0.3.1 - created: "2024-08-09T17:23:25.890104631+05:30" + created: "2024-08-24T14:39:26.019055719+05:30" dependencies: - condition: vector.enabled name: vector @@ -632,7 +655,7 @@ entries: version: 0.3.1 - apiVersion: v2 appVersion: v0.3.0 - created: "2024-08-09T17:23:25.887921136+05:30" + created: "2024-08-24T14:39:26.011667402+05:30" description: Helm chart for Parseable Server digest: ff30739229b727dc637f62fd4481c886a6080ce4556bae10cafe7642ddcfd937 name: parseable @@ -642,7 +665,7 @@ entries: version: 0.3.0 - apiVersion: v2 appVersion: v0.2.2 - created: "2024-08-09T17:23:25.887744822+05:30" + created: "2024-08-24T14:39:26.010981864+05:30" description: Helm chart for Parseable Server digest: 477d0dc2f0c07d4f4c32e105d4bdd70c71113add5c2a75ac5f1cb42aa0276db7 name: parseable @@ -652,7 +675,7 @@ entries: version: 0.2.2 - apiVersion: v2 appVersion: v0.2.1 - created: "2024-08-09T17:23:25.887575199+05:30" + created: "2024-08-24T14:39:26.010362068+05:30" description: Helm chart for Parseable Server digest: 84826fcd1b4c579f301569f43b0309c07e8082bad76f5cdd25f86e86ca2e8192 name: parseable @@ -662,7 +685,7 @@ entries: version: 0.2.1 - apiVersion: v2 appVersion: v0.2.0 - created: "2024-08-09T17:23:25.8874199+05:30" + created: "2024-08-24T14:39:26.009808831+05:30" description: Helm chart for Parseable Server digest: 7a759f7f9809f3935cba685e904c021a0b645f217f4e45b9be185900c467edff name: parseable @@ -672,7 +695,7 @@ entries: version: 0.2.0 - apiVersion: v2 appVersion: v0.1.1 - created: "2024-08-09T17:23:25.887270282+05:30" + created: "2024-08-24T14:39:26.009275683+05:30" description: Helm chart for Parseable Server digest: 37993cf392f662ec7b1fbfc9a2ba00ec906d98723e38f3c91ff1daca97c3d0b3 name: parseable @@ -682,7 +705,7 @@ entries: version: 0.1.1 - apiVersion: v2 appVersion: v0.1.0 - created: "2024-08-09T17:23:25.887120079+05:30" + created: "2024-08-24T14:39:26.008744905+05:30" description: Helm chart for Parseable Server digest: 1d580d072af8d6b1ebcbfee31c2e16c907d08db754780f913b5f0032b403789b name: parseable @@ -692,7 +715,7 @@ entries: version: 0.1.0 - apiVersion: v2 appVersion: v0.0.8 - created: "2024-08-09T17:23:25.886958706+05:30" + created: "2024-08-24T14:39:26.008210787+05:30" description: Helm chart for Parseable Server digest: c805254ffa634f96ecec448bcfff9973339aa9487dd8199b21b17b79a4de9345 name: parseable @@ -702,7 +725,7 @@ entries: version: 0.0.8 - apiVersion: v2 appVersion: v0.0.7 - created: "2024-08-09T17:23:25.886774528+05:30" + created: "2024-08-24T14:39:26.007694623+05:30" description: Helm chart for Parseable Server digest: c591f617ed1fe820bb2c72a4c976a78126f1d1095d552daa07c4700f46c4708a name: parseable @@ -712,7 +735,7 @@ entries: version: 0.0.7 - apiVersion: v2 appVersion: v0.0.6 - created: "2024-08-09T17:23:25.886458047+05:30" + created: "2024-08-24T14:39:26.00708371+05:30" description: Helm chart for Parseable Server digest: f9ae56a6fcd6a59e7bee0436200ddbedeb74ade6073deb435b8fcbaf08dda795 name: parseable @@ -722,7 +745,7 @@ entries: version: 0.0.6 - apiVersion: v2 appVersion: v0.0.5 - created: "2024-08-09T17:23:25.885636578+05:30" + created: "2024-08-24T14:39:26.006451871+05:30" description: Helm chart for Parseable Server digest: 4d6b08a064fba36e16feeb820b77e1e8e60fb6de48dbf7ec8410d03d10c26ad0 name: parseable @@ -732,7 +755,7 @@ entries: version: 0.0.5 - apiVersion: v2 appVersion: v0.0.2 - created: "2024-08-09T17:23:25.885336565+05:30" + created: "2024-08-24T14:39:26.005643644+05:30" description: Helm chart for Parseable Server digest: 38a0a3e4c498afbbcc76ebfcb9cb598fa2ca843a53cc93b3cb4f135b85c10844 name: parseable @@ -742,7 +765,7 @@ entries: version: 0.0.2 - apiVersion: v2 appVersion: v0.0.1 - created: "2024-08-09T17:23:25.884994837+05:30" + created: "2024-08-24T14:39:26.004974696+05:30" description: Helm chart for Parseable Server digest: 1f1142db092b9620ee38bb2294ccbb1c17f807b33bf56da43816af7fe89f301e name: parseable @@ -771,4 +794,4 @@ entries: urls: - https://charts.parseable.io/helm-releases/parseable-operator-0.0.1.tgz version: 0.0.1 -generated: "2024-08-09T17:23:25.884589459+05:30" +generated: "2024-08-24T14:39:26.003897997+05:30"