Handle file uploads

[Rich-Harris]

Currently, the body parser bails on multipart forms that include files

[benmccann]

It looks like the way you'd do this in Sapper is to add something like Multer or Formidable as a middleware which would parse the incoming request and stick the files on the request object. Probably the way to do this would be to expose a way to add middleware, so that users can utilize these packages, so I wonder if this is essentially a duplicate of #334. (side note, I wonder if we should have adapter-polka and adapter-express instead of adapter-node)

I was trying to figure out how Netlify, Lambda, etc. handle this. All the examples I see directly upload to S3 from the browser. I couldn't find how you'd handle a form upload inside a Lambda, so I guess that's something that's generally not supported

[thgh]

Here is an example of file uploads to lambda: https://github.com/mpxr/lambda-file-upload/blob/master/handler.js

I may be biased, but I think serverless APIs typically don't use global middleware and instead have several awaits (one per middleware) at the start of the function. See https://nextjs.org/docs/api-routes/api-middlewares#connectexpress-middleware-support

[TomStrepsil]

For what it's worth, confirmed that this was working fine with sapper + Formidable but not working with sveltekit.

[axmad386]

I think hooks is enough for handling Middleware, except the request (ServerRequest) is little bit different from polka or express. Multer require request.pipe and unpipe to handle File Upload. Formidable require request.on which is missing from request object on handle hooks. Cmiiw

[benmccann]

You'd have to convert from SvelteKit request to Express request first before calling the middleware

[axmad386]

You'd have to convert from SvelteKit request to Express request first before calling the middleware

That's exactly I want, but how. Any references?

[vonadz]

Would it be possible to comment out this section

kit/packages/kit/src/runtime/server/parse_body/index.js

Lines 92 to 95 in e7f6fef

	if (directives.filename) {
	// TODO we probably don't want to do this automatically
	throw new Error('File upload is not yet implemented');
	}

for now? This will give users the chance to implement their own solutions in the handle hook.

EDIT: If anyone is trying to find a solution for this, or a work around, here is what I did for an app that will work with the adapter-node:

1. Comment out the throw Error part of the code I linked above in your node_module @sveltejs/kit (I found this by just using
   grep -rin 'File upload is not yet implemented' ./node_modules/@sveltejs/)
2. Install busboy
3. Add the code below to your hooks file

import busboy from 'busboy'
import type { Handle } from '@sveltejs/kit'
export const handle: Handle = async ({ request, render }) => {
  console.log('in handle')
  console.log(request)
  if (request.path === 'your/path') {
    const files = {}
    await new Promise((resolve, reject) => {
      const bb = new busboy({
        headers: { 'content-type': request.headers['content-type'] },
      })
      bb.on('file', (fieldname, file, filename, encoding, mimetype) => {
        console.log(
          'File [%s]: filename=%j; encoding=%j; mimetype=%j',
          fieldname,
          filename,
          encoding,
          mimetype
        )
        console.log(file)
        const buffers = []
        files[fieldname] = {
          filename,
          encoding,
          mimetype,
        }
        file
          .on('data', (fileData) => {
            console.log('File [%s] got %d bytes', fieldname, fileData.length)
            buffers.push(fileData) //this adds files as buffers, which can be hard on memory. You can also write to a file using streams here.
          })
          .on('end', () => {
            console.log('File [%s] Finished', fieldname)
            const buffer = Buffer.concat(buffers)
            files[fieldname]['file'] = buffer
          })
      })
        .on('field', (fieldname, val) => {
          console.log('Field [%s]: value: %j', fieldname, val)
          request.body[fieldname] = val
        })
        .on('finish', () => {
          console.log('Done parsing form!')
          request.body['files'] = files
          resolve(request.body)
        })
        .on('error', (err) => {
          console.log('failed', err)
          reject(err)
        })

      bb.end(request.rawBody)
    })
  }
  const response = await render(request)
  return {
    ...response,
    headers: {
      ...response.headers,
    },
  }
}

EDIT 2: so I can't seem to get the above code to actually write nonbroken files. I've tried a couple of different methods, including substituting busboy for something like aws-lambda-multipart-parser, which gave files that were the correct size, but still corrupted or something. Maybe there is an issue in encoding/decoding processes for the rawBody that happens here:

kit/packages/kit/src/core/http/index.js

Lines 18 to 29 in f3ef93d

	if (!isNaN(length)) {
	data = new Uint8Array(length);
	let i = 0;
	req.on('data', (chunk) => {
	// TODO maybe there's a simpler way to copy data between buffers?
	for (let j = 0; j < chunk.length; j += 1) {
	data[i++] = chunk[j];
	}
	});
	} else {

EDIT 3: so changing the Buffer type for content-type multipart/form-data in the above code to 'ascii' gives me files of the exact size that are sent, but they're still corrupted for some reason. Anyone know what I might be missing here?

[MTyson]

I just did this in Sapper, and I'll throw out my thoughts in case it might help.

I used formidable and filepond. Filepond sends the chunks, and I used formidable to write a temp file to disk (via node fs). Upon submitting the form, a unique id is used to retrieve the file and write it to its permanent home (mongodb in this case).

In a pure serverless environment, you could override formidable and handle the chunks yourself, writing them to memory, and do the same process. (This assumes we can import formidable into sveltekit).

In the end, there should be a simple way to interact with the filesystem if there is one at hand.

[vonadz]

I just did this in Sapper, and I'll throw out my thoughts in case it might help.

I used formidable and filepond. Filepond sends the chunks, and I used formidable to write a temp file to disk (via node fs). Upon submitting the form, a unique id is used to retrieve the file and write it to its permanent home (mongodb in this case).

In a pure serverless environment, you could override formidable and handle the chunks yourself, writing them to memory, and do the same process. (This assumes we can import formidable into sveltekit).

In the end, there should be a simple way to interact with the filesystem if there is one at hand.

Yeah I played around with using formidable, but I don't think it's a very nice solution. From my understanding, the handle hook would be the natural place for handling fileupload. The issue is that the request in the handle hook isn't a stream of chunks, like is normally expected for something like formidable or express-fileupload, but an object with a rawBody that is the complete stream in string format. The reason I was using busboy is because it can parse out form data out of a string, both fields and files.

[MTyson]

handle hook isn't a stream of chunks

Interesting. So something in sveltekit itself is handling the chunks and marshalling them into a string of bits?

I wonder if that could pose a limitation in cases where one wanted to stream the chunks directly to a datastore...?

[juanmait]

handle hook isn't a stream of chunks

Interesting. So something in sveltekit itself is handling the chunks and marshalling them into a string of bits?

I wonder if that could pose a limitation in cases where one wanted to stream the chunks directly to a datastore...?

I'm just looking for that. I'm working with streaming in the browser side and now reached the point of wanting to stream the result of the client pipeline to the server. Seems like there is no way to access the raw request in sveltekit endpoints. I was planning to use cloudflare workers but first giving it a try using nodejs.

I would vote for having a way to access the plain raw request object on whatever the platform you use. I guess this would be the job of every adapter.