Skip to content

Add errors only self-hosted infrastructure #3190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 22, 2024
+91 −36
Merged

Add errors only self-hosted infrastructure #3190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
dfd4235
first pass
hubertdeng123 Jul 9, 2024
36e286b
change to feature complete version of self-hosted
hubertdeng123 Jul 9, 2024
92f11de
add errors only option to config
hubertdeng123 Jul 9, 2024
005c493
check the output of docker compose up
hubertdeng123 Jul 9, 2024
6244c9b
remove docker compose up in test yml
hubertdeng123 Jul 9, 2024
414ba30
much cleaner way of setting up docker compose file
hubertdeng123 Jul 10, 2024
417886f
reduce depends on clauses
hubertdeng123 Jul 15, 2024
a7a0908
get rid of unused anchor
hubertdeng123 Jul 15, 2024
3338396
merge master
hubertdeng123 Jul 16, 2024
251091e
fix
hubertdeng123 Jul 16, 2024
a0333f2
consolidate flags
hubertdeng123 Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
COMPOSE_PROJECT_NAME=sentry-self-hosted
COMPOSE_PROFILES=feature-complete
SENTRY_EVENT_RETENTION_DAYS=90
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
Expand Down
123 changes: 87 additions & 36 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,24 +36,8 @@ x-sentry-defaults: &sentry_defaults
<<: *depends_on-default
snuba-api:
<<: *depends_on-default
snuba-errors-consumer:
Copy link
Member Author

@hubertdeng123 hubertdeng123 Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not needed to bring up the web container and other ingest consumers

<<: *depends_on-default
snuba-outcomes-consumer:
<<: *depends_on-default
snuba-outcomes-billing-consumer:
<<: *depends_on-default
snuba-transactions-consumer:
<<: *depends_on-default
snuba-subscription-consumer-events:
<<: *depends_on-default
snuba-subscription-consumer-transactions:
<<: *depends_on-default
snuba-replacer:
<<: *depends_on-default
symbolicator:
<<: *depends_on-default
vroom:
<<: *depends_on-default
entrypoint: "/etc/sentry/entrypoint.sh"
command: ["run", "web"]
environment:
Expand All @@ -71,6 +55,7 @@ x-sentry-defaults: &sentry_defaults
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH_ENV_VAR: *ca_bundle
# Leaving the value empty to just pass whatever is set
# on the host system (or in the .env file)
COMPOSE_PROFILES:
SENTRY_EVENT_RETENTION_DAYS:
SENTRY_MAIL_HOST:
SENTRY_MAX_EXTERNAL_SOURCEMAP_SIZE:
Expand Down Expand Up @@ -268,55 +253,84 @@ services:
snuba-outcomes-billing-consumer:
<<: *snuba_defaults
command: rust-consumer --storage outcomes_raw --consumer-group snuba-consumers --auto-offset-reset=earliest --max-batch-time-ms 750 --no-strict-offset-reset --raw-events-topic outcomes-billing
snuba-group-attributes-consumer:
<<: *snuba_defaults
command: rust-consumer --storage group_attributes --consumer-group snuba-group-attributes-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-replacer:
<<: *snuba_defaults
command: replacer --storage errors --auto-offset-reset=latest --no-strict-offset-reset
snuba-subscription-consumer-events:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset events --entity events --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-events-subscriptions-consumers --followed-consumer-group=snuba-consumers --schedule-ttl=60 --stale-threshold-seconds=900
#############################################
## Feature Complete Sentry Snuba Consumers ##
#############################################
# Kafka consumer responsible for feeding transactions data into Clickhouse
snuba-transactions-consumer:
<<: *snuba_defaults
command: rust-consumer --storage transactions --consumer-group transactions_group --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-replays-consumer:
<<: *snuba_defaults
command: rust-consumer --storage replays --consumer-group snuba-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-issue-occurrence-consumer:
<<: *snuba_defaults
command: rust-consumer --storage search_issues --consumer-group generic_events_group --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-metrics-consumer:
<<: *snuba_defaults
command: rust-consumer --storage metrics_raw --consumer-group snuba-metrics-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-group-attributes-consumer:
profiles:
- feature-complete
snuba-subscription-consumer-transactions:
<<: *snuba_defaults
command: rust-consumer --storage group_attributes --consumer-group snuba-group-attributes-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
command: subscriptions-scheduler-executor --dataset transactions --entity transactions --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-transactions-subscriptions-consumers --followed-consumer-group=transactions_group --schedule-ttl=60 --stale-threshold-seconds=900
profiles:
- feature-complete
snuba-subscription-consumer-metrics:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset metrics --entity metrics_sets --entity metrics_counters --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-metrics-subscriptions-consumers --followed-consumer-group=snuba-metrics-consumers --schedule-ttl=60 --stale-threshold-seconds=900
profiles:
- feature-complete
snuba-generic-metrics-distributions-consumer:
<<: *snuba_defaults
command: rust-consumer --storage generic_metrics_distributions_raw --consumer-group snuba-gen-metrics-distributions-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-generic-metrics-sets-consumer:
<<: *snuba_defaults
command: rust-consumer --storage generic_metrics_sets_raw --consumer-group snuba-gen-metrics-sets-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-generic-metrics-counters-consumer:
<<: *snuba_defaults
command: rust-consumer --storage generic_metrics_counters_raw --consumer-group snuba-gen-metrics-counters-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
profiles:
- feature-complete
snuba-generic-metrics-gauges-consumer:
<<: *snuba_defaults
command: rust-consumer --storage generic_metrics_gauges_raw --consumer-group snuba-gen-metrics-gauges-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-replacer:
<<: *snuba_defaults
command: replacer --storage errors --auto-offset-reset=latest --no-strict-offset-reset
snuba-subscription-consumer-events:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset events --entity events --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-events-subscriptions-consumers --followed-consumer-group=snuba-consumers --schedule-ttl=60 --stale-threshold-seconds=900
snuba-subscription-consumer-transactions:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset transactions --entity transactions --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-transactions-subscriptions-consumers --followed-consumer-group=transactions_group --schedule-ttl=60 --stale-threshold-seconds=900
snuba-subscription-consumer-metrics:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset metrics --entity metrics_sets --entity metrics_counters --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-metrics-subscriptions-consumers --followed-consumer-group=snuba-metrics-consumers --schedule-ttl=60 --stale-threshold-seconds=900
profiles:
- feature-complete
snuba-profiling-profiles-consumer:
<<: *snuba_defaults
command: rust-consumer --storage profiles --consumer-group snuba-consumers --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset
profiles:
- feature-complete
snuba-profiling-functions-consumer:
<<: *snuba_defaults
command: rust-consumer --storage functions_raw --consumer-group snuba-consumers --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset
profiles:
- feature-complete
snuba-spans-consumer:
<<: *snuba_defaults
command: rust-consumer --storage spans --consumer-group snuba-spans-consumers --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset
profiles:
- feature-complete
symbolicator:
<<: *restart_policy
image: "$SYMBOLICATOR_IMAGE"
Expand Down Expand Up @@ -363,57 +377,90 @@ services:
attachments-consumer:
<<: *sentry_defaults
command: run consumer ingest-attachments --consumer-group ingest-consumer
post-process-forwarder-errors:
<<: *sentry_defaults
command: run consumer --no-strict-offset-reset post-process-forwarder-errors --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-commit-log --synchronize-commit-group=snuba-consumers
subscription-consumer-events:
<<: *sentry_defaults
command: run consumer events-subscription-results --consumer-group query-subscription-consumer
##############################################
## Feature Complete Sentry Ingest Consumers ##
##############################################
transactions-consumer:
<<: *sentry_defaults
command: run consumer ingest-transactions --consumer-group ingest-consumer
profiles:
- feature-complete
metrics-consumer:
<<: *sentry_defaults
command: run consumer ingest-metrics --consumer-group metrics-consumer
profiles:
- feature-complete
generic-metrics-consumer:
<<: *sentry_defaults
command: run consumer ingest-generic-metrics --consumer-group generic-metrics-consumer
profiles:
- feature-complete
billing-metrics-consumer:
<<: *sentry_defaults
command: run consumer billing-metrics-consumer --consumer-group billing-metrics-consumer
profiles:
- feature-complete
ingest-replay-recordings:
<<: *sentry_defaults
command: run consumer ingest-replay-recordings --consumer-group ingest-replay-recordings
profiles:
- feature-complete
ingest-occurrences:
<<: *sentry_defaults
command: run consumer ingest-occurrences --consumer-group ingest-occurrences
profiles:
- feature-complete
ingest-profiles:
<<: *sentry_defaults
command: run consumer ingest-profiles --consumer-group ingest-profiles
profiles:
- feature-complete
ingest-monitors:
<<: *sentry_defaults
command: run consumer ingest-monitors --consumer-group ingest-monitors
profiles:
- feature-complete
monitors-clock-tick:
<<: *sentry_defaults
command: run consumer monitors-clock-tick --consumer-group monitors-clock-tick
profiles:
- feature-complete
monitors-clock-tasks:
<<: *sentry_defaults
command: run consumer monitors-clock-tasks --consumer-group monitors-clock-tasks
post-process-forwarder-errors:
<<: *sentry_defaults
command: run consumer --no-strict-offset-reset post-process-forwarder-errors --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-commit-log --synchronize-commit-group=snuba-consumers
profiles:
- feature-complete
post-process-forwarder-transactions:
<<: *sentry_defaults
command: run consumer --no-strict-offset-reset post-process-forwarder-transactions --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-transactions-commit-log --synchronize-commit-group transactions_group
profiles:
- feature-complete
post-process-forwarder-issue-platform:
<<: *sentry_defaults
command: run consumer --no-strict-offset-reset post-process-forwarder-issue-platform --consumer-group post-process-forwarder --synchronize-commit-log-topic=snuba-generic-events-commit-log --synchronize-commit-group generic_events_group
subscription-consumer-events:
<<: *sentry_defaults
command: run consumer events-subscription-results --consumer-group query-subscription-consumer
profiles:
- feature-complete
subscription-consumer-transactions:
<<: *sentry_defaults
command: run consumer transactions-subscription-results --consumer-group query-subscription-consumer
profiles:
- feature-complete
subscription-consumer-metrics:
<<: *sentry_defaults
command: run consumer metrics-subscription-results --consumer-group query-subscription-consumer
profiles:
- feature-complete
subscription-consumer-generic-metrics:
<<: *sentry_defaults
command: run consumer generic-metrics-subscription-results --consumer-group query-subscription-consumer
profiles:
- feature-complete
sentry-cleanup:
<<: *sentry_defaults
image: sentry-cleanup-self-hosted-local
Expand Down Expand Up @@ -469,6 +516,8 @@ services:
depends_on:
kafka:
<<: *depends_on-healthy
profiles:
- feature-complete
vroom-cleanup:
<<: *restart_policy
image: vroom-cleanup-self-hosted-local
Expand All @@ -484,6 +533,8 @@ services:
command: '"0 0 * * * find /var/lib/sentry-profiles -type f -mtime +$SENTRY_EVENT_RETENTION_DAYS -delete"'
volumes:
- sentry-vroom:/var/lib/sentry-profiles
profiles:
- feature-complete

volumes:
# These store application data that should persist across restarts.
Expand Down
3 changes: 3 additions & 0 deletions sentry/sentry.conf.example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,3 +363,6 @@ def get_internal_network():
# https://docs.djangoproject.com/en/4.2/ref/settings/#std-setting-CSRF_TRUSTED_ORIGINS

# CSRF_TRUSTED_ORIGINS = ["https://example.com", "http://127.0.0.1:9000"]

# If you would like to use self-hosted Sentry with only errors enabled, please set this
Copy link
Collaborator

@aldy505 aldy505 Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also define what's "errors only" and what's not? I mean it's clear you'd know what "errors only" means if you previously use Sentry. But I don't think newcomers will instantly understand about what features are available on the "feature complete" profile.

Copy link
Member Author

@hubertdeng123 hubertdeng123 Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I think we should. Going to also need to update docs, which I'll do today.

Copy link

@soupdumplings soupdumplings Jul 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excuse me, does the "errors only" version include the Projects? I reviewed the docs and found that it doesn't have the Projects.

Copy link
Contributor

@stayallive stayallive Jul 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: https://develop.sentry.dev/self-hosted/#errors-only-self-hosted-sentry. But projects are a core concept of Sentry, so yes that is always included.

Copy link

@soupdumplings soupdumplings Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I really need this feature!

Copy link
Member Author

@hubertdeng123 hubertdeng123 Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, projects are always included.

Copy link

@soupdumplings soupdumplings Jul 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://develop.sentry.dev/self-hosted/#errors-only-self-hosted-sentry
Does the "Traces" mentioned in the docs refer to "stack traces" and is it no longer supported under "errors only"?

Copy link
Collaborator

@aldy505 aldy505 Jul 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, traces refers to the submenu inside the "Explore" menu.
image

Stack traces is included on the errors-only since it's a crucial component to the error monitoring mechanism

Copy link

@soupdumplings soupdumplings Jul 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your patient answers. I use self-hosted Sentry as our team's crash analysis platform. It's really powerful and useful. The "errors only" version is just perfect for me.

SENTRY_SELF_HOSTED_ERRORS_ONLY = env("COMPOSE_PROFILES") == "feature-complete"
Copy link
Contributor

@stayallive stayallive Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be env("COMPOSE_PROFILES") == "errors-only" right?

Copy link
Member Author

@hubertdeng123 hubertdeng123 Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right htis is a bug, thanks for raising. I just checked my local sentry.conf.py and it uses SENTRY_SELF_HOSTED_ERRORS_ONLY = env("COMPOSE_PROFILES") != "feature-complete" when I was testing this locally. Opened #3220

Copy link
Contributor

@stayallive stayallive Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I wasn't 100% sure but I just also debugged a self-hosted instance very bravely using nightly and was caught off guard by this 😅 Thanks for the quick PR!