feat: re-enable time-partitioning #1267
Conversation
WalkthroughThis pull request refactors how time partitioning is represented and processed in event structures. The nullable Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant HTTP as HTTP Ingest Handler
participant EventProc as Event Processor
participant TimeGen as Utc::now
Client->>HTTP: Submit event log
HTTP->>EventProc: Call push_logs_unchecked (with is_time_partitioned = false)
EventProc->>TimeGen: If is_time_partitioned is true, request current timestamp
TimeGen-->>EventProc: Return current timestamp
EventProc-->>HTTP: Processed event returned
HTTP-->>Client: Response sent
sequenceDiagram
participant Client as Client
participant Parser as Parseable Module
participant StreamMgr as Stream Manager
Client->>Parser: Request stream creation/update
Parser->>Parser: Validate time_partition_limit and custom_partition
Parser->>StreamMgr: Call create_stream with time_partition_in_days (if applicable)
StreamMgr-->>Parser: Stream creation/update response
Parser-->>Client: Return outcome of stream creation/update
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (10)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/parseable/streams.rs (1)
147-148: Timestamp source changed from parsed_timestamp to current timeThe code now uses
Utc::now()instead of the providedparsed_timestampwhen creating the time range for theDiskWriter. This change is part of re-enabling time-partitioning and shifts from event time to ingestion time for determining partition boundaries.Consider adding a comment explaining the reasoning behind using the current time rather than the parsed timestamp to make the intent clearer to future developers.
+ // Use current time for partitioning to ensure consistent partition boundaries + // based on ingestion time rather than event timestamps let range = TimeRange::granularity_range(Utc::now(), OBJECT_STORE_DATA_GRANULARITY);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/parseable/streams.rs(1 hunks)
🧰 Additional context used
🧬 Code Definitions (1)
src/parseable/streams.rs (1)
src/utils/time.rs (1)
granularity_range(267-282)
⏰ Context from checks skipped due to timeout of 90000ms (10)
- GitHub Check: Build Default aarch64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-apple-darwin
- GitHub Check: Build Default x86_64-pc-windows-msvc
- GitHub Check: Build Default aarch64-apple-darwin
- GitHub Check: Build Kafka x86_64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-unknown-linux-gnu
- GitHub Check: Build Kafka aarch64-apple-darwin
- GitHub Check: Quest Smoke and Load Tests for Distributed deployments
- GitHub Check: coverage
- GitHub Check: Quest Smoke and Load Tests for Standalone deployments
Signed-off-by: Devdutt Shenoi <[email protected]>
|
CLA Assistant Lite bot: I have read the CLA Document and I hereby sign the CLA Devdutt Shenoi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. |
2 participants
Fixes #XXXX.
Description
This PR has:
Summary by CodeRabbit