feat(indexer): Add cross organisation methods for the string indexer #45076
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Scope: Backend
RaduW
force-pushed
the
feat/indexer/add-cross-org-support
branch
from
1fb9740 to
f1e0f8e
Compare
andriisoldatenko
approved these changes
Feb 27, 2023
5 tasks
andriisoldatenko
pushed a commit
that referenced
this pull request
Feb 28, 2023
#42939) This PR implements prioritize by project bias. In detail: We run celery task every 24 at 8:00AM (UTC randomly selected) for every ORG (we call it *prioritise by project snuba query* ) and all projects inside this org, and for a given combination of org and projects run an adjustment model to recalculate sample rates if necessary. Then we cache sample rate using redis cluster -> `SENTRY_DYNAMIC_SAMPLING_RULES_REDIS_CLUSTER` using this pattern for key: `f"ds::o:{org_id}:p:{project_id}:prioritise_projects"`. When relay fetches `projectconfig` endpoint we run `generate_rules` functions to generate all dynamic sampling biases, so and we check if we have adjusted sample rate for this project in the cache, so we apply it as **uniform bias**, otherwise we use default one. Regarding *prioritize by project snuba query* is cross org snuba query that utilizes a new generic counter metric, which was introduced in [relay]( getsentry/relay#1734) `c:transactions/count_per_root_project@none`. TODO: - [x] Provision infrastructure to run clickhouse clusters for the counters tables. This is primarily dependent on ops - [x] Start running the snuba consumers to read and write to the counters table. SnS can work on this - [x] Add unit-tests; - [x] Update snuba query using new metric - [x] Hide behind feature flag related PRs: - Implement new metric in relay: getsentry/relay#1734 - Add org generic counters [TET-695] getsentry/snuba#3708 - Introduce new storages for counters in snuba getsentry/snuba#3679 - Add feature flag: https://github.com/getsentry/getsentry/pull/9323 - Add cross organization methods for the string indexer #45076 #45076 [TET-695]: https://getsentry.atlassian.net/browse/TET-695?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com> Co-authored-by: Nar Saynorath <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds methods that work with global (cross organisation) strings.
The methods are meant to be used only for cross organisation queries.