Skip to content

update thanos and add block_ids_fetcher to bucketindex #5681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Nov 28, 2023
Merged

update thanos and add block_ids_fetcher to bucketindex #5681

merged 8 commits into from
Nov 28, 2023

Conversation

@wenxu1024
Copy link
Contributor

@wenxu1024 wenxu1024 commented Nov 27, 2023 •
edited
Loading

What this PR does:

  1. Update Thanos to latest version.
  2. Added bucket-index-ids-fetcher to allow compactor to use bucket-index to get the list of active and partial blocks. This could decrease the meta data syncer time from more than 10m to less than 1m for large tenants.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24
yeya24 reviewed Nov 28, 2023
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I like this pr. Let's make sure we have enough tests coverage for the new block IDs fetcher and the compactor itself.
Unfortunately we don't have integration tests for compactor rn. We should have that at some point.

pkg/compactor/bucket-index-block-ids-fetcher.go Outdated
}
}

func (f *BucketIndexBlockIDsFetcher) GetActiveAndPartialBlockIDs(ctx context.Context, ch chan<- ulid.ULID) (partialBlocks map[ulid.ULID]bool, err error) {
Copy link
Contributor

@yeya24 yeya24 Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make sure we have unit tests to cover this

Copy link
Contributor Author

@wenxu1024 wenxu1024 Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added unit test.

@yeya24
Copy link
Contributor

yeya24 commented Nov 28, 2023

Btw do we still need #5641?

@wenxu1024
Copy link
Contributor Author

wenxu1024 commented Nov 28, 2023

Btw do we still need #5641?

Thanks. We can close that one.

@wenxu1024 wenxu1024 changed the title update thanos and add bucket-index-ids-fetcher to compactor update thanos and add bucket-index-ids-fetcher to bucketindex Nov 28, 2023
@wenxu1024 wenxu1024 changed the title update thanos and add bucket-index-ids-fetcher to bucketindex update thanos and add block_ids_fetcher to bucketindex Nov 28, 2023
yeya24
yeya24 reviewed Nov 28, 2023
View reviewed changes
danielblando
danielblando reviewed Nov 28, 2023
View reviewed changes
@wenxu1024
remove TODO
52c8a50 
Signed-off-by: Wen Xu <[email protected]>
yeya24
yeya24 approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 Thanks!

danielblando
danielblando approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@danielblando danielblando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change

@yeya24 yeya24 merged commit 7f8d194 into cortexproject:master Nov 28, 2023
@wenxu1024 wenxu1024 mentioned this pull request Dec 19, 2023
Merged
3 tasks
@yeya24 yeya24 mentioned this pull request Mar 25, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danielblando danielblando danielblando approved these changes

@yeya24 yeya24 yeya24 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wenxu1024 @yeya24 @danielblando