Receiving the "Failed to submit event" for every event sent while the event s are logged

[kirlev]

Describe the bug
As described here: https://forum.sentry.io/t/unable-to-record-event-with-remote-sentry-server-sentry-error-the-server-responded-with-status-429/13553

Seeing this error happening for every event AFAIK but the event is logged in and visible in the Sentry dashboard.

I saw other complain about this issue, for example - https://forum.sentry.io/t/sending-events-from-application-returns-error-but-events-are-saved-in-sentry/13115

The error:

Unable to record event with remote Sentry server (Sentry::Error - the server responded with status 429
body: {"detail":"event rejected due to rate limit"}):
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/transport/http_transport.rb:29:in `rescue in send_data'
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/transport/http_transport.rb:15:in `send_data'
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/transport.rb:31:in `send_event'
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/client.rb:95:in `send_event'
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/client.rb:47:in `block in capture_event'
/usr/local/bundle/ruby/2.6.0/gems/sentry-ruby-core-4.2.2/lib/sentry/background_worker.rb:33:in `block in perform'
/usr/local/bundle/ruby/2.6.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/usr/local/bundle/ruby/2.6.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/usr/local/bundle/ruby/2.6.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/usr/local/bundle/ruby/2.6.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/usr/local/bundle/ruby/2.6.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'

To Reproduce
I followed the instructions in the documentation here: https://docs.sentry.io/platforms/ruby/

Expected behavior
report event without errors

Actual behavior
the events are reported and logged but exceptions are thrown.

Environment

• Ruby Version: 2.6.6
• SDK Version:

• sentry-rails (4.2.2)
• sentry-ruby (4.2.2)
• sentry-ruby-core (4.2.2)

• Integration Versions (if any):
• Rails 5.2.3

Raven Config

This is not necessary but could be helpful.

Sentry.init do |config|
  config.dsn = settings.fetch(:dsn)
  config.breadcrumbs_logger = %i[active_support_logger]
  config.enabled_environments = %w[production staging]
  config.traces_sample_rate = 1.0
  config.release = ENV.fetch('GIT_COMMIT', nil)
  config.excluded_exceptions +=
    %w[
      ActionController::RoutingError
      AbstractController::ActionNotFound
      ActiveRecord::RecordNotFound
      Errors::NotFoundError
    ]

  config.send_default_pii = true if Rails.env.staging?
end

[st0012]

this is because some events (perhaps tracing events) were rejected by Sentry due to rate limiting. so it's not a bug in the SDK that raises the exception.
that being said, in the upcoming 4.4.0 version, those rate limiting responses will be handled more elegantly. the SDK will also stop sending events for a certain amount of time if it receives such responses. related PR: #1336

[kirlev]

Thanks for the quick response @st0012 !

I don't have rate limiting for that specific project so why would Sentry reject the events?
In any case, seems that removing this line: config.traces_sample_rate = 1.0 from the configuration block solved the issue.

[st0012]

Sentry rate limits all types of incoming events to avoid huge spikes from consuming quota too quickly and cause large pressure on the server.

config.traces_sample_rate = 1.0 means sending every tracing events to Sentry. so if your application has 10k requests per minute, it'll also send 10k tracing events to Sentry. and many of them must've been rejected due to rate limiting, which then caused the logs you see.

[rhcarvalho]

@st0012 since #1336 is merged, can we close this issue or is there any more work tracked here?

[st0012]

yeah I think it can be closed now