- 
                Notifications
    You must be signed in to change notification settings 
- Fork 482
Convert persist_source to a regular non-async operator #17252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert persist_source to a regular non-async operator #17252
Conversation
3f4ef85    to
    6445833      
    Compare
  
    | I'm supportive of the overall direction, but the original behavior of  | 
| I can bring back the  | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will defer to storage folks regarding the details of the operator code; at least upon a comparison of the old and new versions, it seems to me that the logic has been kept unharmed.
Regarding the changes to ConsolidateBuffer, if we now introduce the function give_at, would it make sense to have a variation of new that does not require the output port to be specified, with an explicit Option being passed in? I am a bit worried of misuses where one gives a port in a multi-output operator, but calls give_at by mistake. With some extra checking, the implementation could potentially catch these problems?
Signed-off-by: Moritz Hoffmann <[email protected]>
This requires a change to the ConsolidateBuffer, which previously only worked with `CapabilityRef` instead of actual capabilities. The Ref type can be narrowed down to specific outputs, but owned capabilities are set to a specific output. Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
6445833    to
    d5eb6f1      
    Compare
  
    Signed-off-by: Moritz Hoffmann <[email protected]>
| I updated the PR to use the  | 
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
persist_source change LGTM! I didn't look at the buffer.rs change. sorry this has been such a saga!
Signed-off-by: Moritz Hoffmann <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Took another look at give_at; it was actually fine, so this LGTM!
Convert the
persist_sourcedecode_and_mfpoperator to a plain Timely operator without usingasync. This avoids maintaining Timely state across yields and seems simpler to reason about in general.This also takes the intended changes from #17156 to compact the output in-place using the
ConsolidateBuffer. This way, we don't have to worry about large allocations or spending significant time in consolidating updates.Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way) and therefore is tagged with aT-protolabel.companion cloud PR to account for those changes that is tagged with
the release-blocker label (example).