Handling streaming request/response bodies #3419
Description
Describe the problem
As of #3384, hooks and endpoints receive a standard Request object, which among other things makes it possible to POST files:
// handles submission from a <form enctype="multipart/form-data">
export async function post({ request }) {
const body = await request.formData();
const file = await body.get('myfile');
// ...
}The file is buffered in memory, however, which means there's a practical limit on how large it can be.
Similarly, there's no good way to read a stream of binary data, or respond with a stream.
Describe the proposed solution
We need to solve this at a high level and at a low level. At the high level, it would be good to have utility functions for dealing with multipart form data specifically — something like this, perhaps:
import { multipart } from '$app/somewhere';
export async function post({ request }) {
for await (const part of multipart(request)) {
if (part.filename) {
const uploader = create_s3_uploader(BUCKET, part.filename);
for (const chunk of part) {
uploader.write(chunk);
}
}
}
return { status: 201 };
}For octet streams it might look like this:
import { stream } from '$app/somewhere';
export async function post({ request }) {
for await (const chunk of stream(request)) {
// ...
}
return { status: 201 };
}At the low level, it should be possible to create your own stream reader and return your own ReadableStream bodies:
export async function post({ request }) {
const reader = request.body.getReader();
let chunk;
while (chunk = await reader.read()) {
if (chunk.done) break;
await do_something_with(chunk.value);
}
return {
body: new ReadableStream({
start(controller) {
// ...
},
cancel() {
// ...
}
})
};
}Unfortunately there's a wrinkle: ReadableStream isn't available in older versions of Node, and the Request and Response objects (as polyfilled by node-fetch) use node streams instead of ReadableStream. It may therefore be necessary to polyfill ReadableStream during development (and in adapter-node, and lambdas) and convert between that and node streams. (On the request object, this would probably mean creating a Proxy that replaces body with a ReadableStream with the same contents as the node stream. Hopefully this is actually possible.)
Alternatives considered
No response
Importance
would make my life easier
Additional Information
No response