Fresh install error: AssertionError sentry.eventstream.kafka.backend in _build_consumer

[ssjunior]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Version

Sentry 22.10.0.dev0

Steps to Reproduce

Do a fresh install using debian bulls-eye.

git clone https://github.com/getsentry/self-hosted.git sentry
cd sentry
./install.sh

go to webpage at port 9000

Expected Result

No issues reported after fresh install

Actual Result

Installed 3 times, same result. After a few seconds an issue is reported:

The issues continue to be reported growing the number of incidents.

[emmatyping]

Hm, sorry you are running into this issue. Are there any more details if you click through to the issue? I cannot reproduce this. You might also want to check if there is any hints in the docker compose logs.

[ssjunior]

Hi @ethanhs,

Checking the logs of docker compose I can see several exceptions like this:

18:55:20 [INFO] sentry.eventstream.kafka.backend: Starting post process forwarder to consume all messages
sentry-self-hosted-post-process-forwarder-1                    | /usr/local/lib/python3.8/site-packages/sentry/runner/initializer.py:563: DeprecatedSettingWarning: The SENTRY_URL_PREFIX setting is deprecated. Please use SENTRY_OPTIONS['system.url-prefix'] instead.
sentry-self-hosted-post-process-forwarder-1                    |   warnings.warn(DeprecatedSettingWarning(old, "SENTRY_OPTIONS['%s']" % new))

Maybe this is related to The SENTRY_URL_PREFIX setting is deprecated issue (getsentry/self-hosted#1309) ?

[ssjunior]

The rest of traceback:

 Traceback (most recent call last):
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-post-process-forwarder-1                    |     sys.exit(main())
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 188, in main
sentry-self-hosted-post-process-forwarder-1                    |     func(**kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-post-process-forwarder-1                    |     return self.main(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-post-process-forwarder-1                    |     rv = self.invoke(ctx)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-1                    |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-1                    |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-post-process-forwarder-1                    |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-1                    |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 363, in post_process_forwarder
sentry-self-hosted-post-process-forwarder-1                    |     eventstream.run_post_process_forwarder(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/services.py", line 127, in <lambda>
sentry-self-hosted-post-process-forwarder-1                    |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 288, in run_post_process_forwarder
sentry-self-hosted-post-process-forwarder-1                    |     self.run_batched_consumer(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 254, in run_batched_consumer
sentry-self-hosted-post-process-forwarder-1                    |     consumer = self._build_consumer(
sentry-self-hosted-post-process-forwarder-1                    |   File "/usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py", line 218, in _build_consumer
sentry-self-hosted-post-process-forwarder-1                    |     assert self.topic == self.transactions_topic
sentry-self-hosted-post-process-forwarder-1                    | AssertionError

[lbogdan]

Getting this too, with latest master, on a minimal Ubuntu Server 22.04 install.

Weird thing is that I don't even see that error in Sentry, under the internal project. Do I have to explicitly configure something for that to happen? I created other projects, and I can see their errors.

[lbogdan]

Did a bit of troubleshooting, added a debug log

logger.info(f"topic={self.topic} transactions_topic={self.transactions_topic}")

before the assert

# /usr/local/lib/python3.8/site-packages/sentry/eventstream/kafka/backend.py:218
assert self.topic == self.transactions_topic

and got

10:14:52 [INFO] sentry.eventstream.kafka.backend: Starting post process forwarder to consume all messages
10:14:52 [INFO] sentry.eventstream.kafka.backend: topic=events transactions_topic=transactions

There's a comment above that says:

# Default implementation which processes both errors and transactions
# irrespective of values in the header. This would most likely be the case
# for development environments. For the combined post process forwarder
# to work KAFKA_EVENTS and KAFKA_TRANSACTIONS must be the same currently.

Not sure what that means, but maybe it helps. 🙂

[long2ice]

Same here