avoid unnecessary reservations in std::io::Take::read_to_end #63216
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
r? @bluss (rust_highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
This comment has been minimized.
This comment has been minimized.
oconnor663
force-pushed
the
take_read_to_end
branch
from
bf4e73f to
5b4b937
Compare
|
☔ The latest upstream changes (presumably #62987) made this pull request unmergeable. Please resolve the merge conflicts. |
Prevously the `read_to_end` implementation for `std::io::Take` used its own `limit` as a cap on the `reservation_size`. However, that could still result in an over-allocation like this: 1. Call `reader.take(5).read_to_end(&mut vec)`. 2. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 3. `read` writes 5 bytes. 4. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 5. `read` writes 0 bytes. 6. The read loop ends with `vec` having length 5 and capacity 10. The reservation of 5 bytes was correct for the read at step 2 but unnecessary for the read at step 4. By that second read, `Take::limit` is 0, but the `read_to_end_with_reservation` loop is still using the same `reservation_size` it started with. Solve this by having `read_to_end_with_reservation` take a closure, which lets it get a fresh `reservation_size` for each read. This is an implementation detail which doesn't affect any public API.
oconnor663
force-pushed
the
take_read_to_end
branch
from
5b4b937 to
edb5214
Compare
|
@bluss let me know if I should grab a different reviewer? |
|
r? @sfackler |
|
@bors r+ |
|
📌 Commit edb5214 has been approved by |
bors
added
S-waiting-on-bors
Centril
added a commit
to Centril/rust
that referenced
this pull request
Aug 20, 2019
…kler avoid unnecessary reservations in std::io::Take::read_to_end Prevously the `read_to_end` implementation for `std::io::Take` used its own `limit` as a cap on the `reservation_size`. However, that could still result in an over-allocation like this: 1. Call `reader.take(5).read_to_end(&mut vec)`. 2. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 3. `read` writes 5 bytes. 4. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 5. `read` writes 0 bytes. 6. The read loop ends with `vec` having length 5 and capacity 10. The reservation of 5 bytes was correct for the read at step 2 but unnecessary for the read at step 4. By that second read, `Take::limit` is 0, but the `read_to_end_with_reservation` loop is still using the same `reservation_size` it started with. Solve this by having `read_to_end_with_reservation` take a closure, which lets it get a fresh `reservation_size` for each read. This is an implementation detail which doesn't affect any public API.
Centril
added a commit
to Centril/rust
that referenced
this pull request
Aug 20, 2019
…kler avoid unnecessary reservations in std::io::Take::read_to_end Prevously the `read_to_end` implementation for `std::io::Take` used its own `limit` as a cap on the `reservation_size`. However, that could still result in an over-allocation like this: 1. Call `reader.take(5).read_to_end(&mut vec)`. 2. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 3. `read` writes 5 bytes. 4. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 5. `read` writes 0 bytes. 6. The read loop ends with `vec` having length 5 and capacity 10. The reservation of 5 bytes was correct for the read at step 2 but unnecessary for the read at step 4. By that second read, `Take::limit` is 0, but the `read_to_end_with_reservation` loop is still using the same `reservation_size` it started with. Solve this by having `read_to_end_with_reservation` take a closure, which lets it get a fresh `reservation_size` for each read. This is an implementation detail which doesn't affect any public API.
Centril
added a commit
to Centril/rust
that referenced
this pull request
Aug 20, 2019
…kler avoid unnecessary reservations in std::io::Take::read_to_end Prevously the `read_to_end` implementation for `std::io::Take` used its own `limit` as a cap on the `reservation_size`. However, that could still result in an over-allocation like this: 1. Call `reader.take(5).read_to_end(&mut vec)`. 2. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 3. `read` writes 5 bytes. 4. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 5. `read` writes 0 bytes. 6. The read loop ends with `vec` having length 5 and capacity 10. The reservation of 5 bytes was correct for the read at step 2 but unnecessary for the read at step 4. By that second read, `Take::limit` is 0, but the `read_to_end_with_reservation` loop is still using the same `reservation_size` it started with. Solve this by having `read_to_end_with_reservation` take a closure, which lets it get a fresh `reservation_size` for each read. This is an implementation detail which doesn't affect any public API.
Centril
added a commit
to Centril/rust
that referenced
this pull request
Aug 20, 2019
…kler avoid unnecessary reservations in std::io::Take::read_to_end Prevously the `read_to_end` implementation for `std::io::Take` used its own `limit` as a cap on the `reservation_size`. However, that could still result in an over-allocation like this: 1. Call `reader.take(5).read_to_end(&mut vec)`. 2. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 3. `read` writes 5 bytes. 4. `read_to_end_with_reservation` reserves 5 bytes and calls `read`. 5. `read` writes 0 bytes. 6. The read loop ends with `vec` having length 5 and capacity 10. The reservation of 5 bytes was correct for the read at step 2 but unnecessary for the read at step 4. By that second read, `Take::limit` is 0, but the `read_to_end_with_reservation` loop is still using the same `reservation_size` it started with. Solve this by having `read_to_end_with_reservation` take a closure, which lets it get a fresh `reservation_size` for each read. This is an implementation detail which doesn't affect any public API.
bors
added a commit
that referenced
this pull request
Aug 20, 2019
Rollup of 7 pull requests Successful merges: - #63216 (avoid unnecessary reservations in std::io::Take::read_to_end) - #63265 (Implement `nth_back` for ChunksExactMut) - #63691 (Fix bug in iter::Chain::size_hint) - #63722 (Don't use stage naming in RUSTFLAGS environment variables) - #63723 (Consolidate sigemptyset workarounds) - #63736 (Restore the rustc_plugin crate in the sysroot) - #63743 (Allow git to merge `Cargo.lock`) Failed merges: r? @ghost
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prevously the
read_to_endimplementation forstd::io::Takeused itsown
limitas a cap on thereservation_size. However, that couldstill result in an over-allocation like this:
reader.take(5).read_to_end(&mut vec).read_to_end_with_reservationreserves 5 bytes and callsread.readwrites 5 bytes.read_to_end_with_reservationreserves 5 bytes and callsread.readwrites 0 bytes.vechaving length 5 and capacity 10.The reservation of 5 bytes was correct for the read at step 2 but
unnecessary for the read at step 4. By that second read,
Take::limitis 0, but the
read_to_end_with_reservationloop is still using thesame
reservation_sizeit started with.Solve this by having
read_to_end_with_reservationtake a closure,which lets it get a fresh
reservation_sizefor each read. This is animplementation detail which doesn't affect any public API.