multipart/form-data

A Collection of Interesting Ideas — Last Updated

Repository:
GitHub

Abstract

A web-spec definition of the multipart/form-data format and related algorithms, meant for inclusion in the WHATWG standards.

1. Introduction

An entry list is a list of entries, typically representing the contents of a form. An entry is a tuple consisting of a name (a scalar value string) and a value (either a scalar value string or a File).

These definitions currently live in the XHR spec as part of the FormData definition, but they should probably be moved to HTML as part of HTML issue #6701. [XHR]

The multipart/form-data format provides a way to encode an entry list, which, unlike the application/x-www-form-urlencoded format, can transmit file contents.

2. multipart/form-data serializing

A multipart/form-data boundary is a byte sequence such that:

To generate a multipart/form-data boundary, return an implementation-defined byte sequence which fullfills the conditions for boundaries, such that part of it is randomly generated, with a minimum entropy of 95 bits.

Previous definitions of multipart/form-data required that the boundary associated with a multipart/form-data payload not be present anywhere in the payload other than as a delimiter, although they allow for generating the boundary probabilistically. Since this generation algorithm is separate from a payload, however, it has to specify a minimum entropy instead. [RFC7578] [RFC2046]

If a user agent generates multipart/form-data boundaries with a length of 27 and an entropy of 95 bits, given a payload made specifically to generate collisions with that user agent’s boundaries, the expected length of the payload before a collision is found is well over a yottabyte.


To escape a multipart/form-data name with a string name, an optional encoding encoding (default UTF-8) and an optional boolean isFilename (default false):

  1. If isFilename is true:

    1. Set name to the result of converting name into a scalar value string.

  2. Otherwise:

    1. Assert: name is a scalar value string.

    2. Replace every occurrence of U+000D (CR) not followed by U+000A (LF), and every occurrence of U+000A (LF) not preceded by U+000D (CR), in name, by a string consisting of U+000D (CR) and U+000A (LF).

  3. Let encoded be the result of encoding name with encoding.

  4. Replace every 0x0A (LF) bytes in encoded with the byte sequence `%0A`, 0x0D (CR) with `%0D` and 0x22 (") with `%22`.

  5. Return encoded.

The multipart/form-data chunk serializer takes an entry list entries and an optional encoding encoding (default UTF-8), and returns a tuple of a multipart/form-data boundary and a list of chunks, each of which can be either a byte sequence or a File:

  1. Set encoding to the result of getting an output encoding from encoding.

  2. Let boundary be the result of generating a multipart/form-data boundary.

  3. Let output chunks be an empty list.

  4. For each entry in entries:

    1. Let chunk be a byte sequence containing `--`, followed by boundary, followed by 0x0D 0x0A (CR LF).

    2. Append `Content-Disposition: form-data; name="`, followed by the result of escaping a multipart/form-data name given entry’s name and encoding, followed by 0x22 ("), to chunk.

    3. Let value be entry’s value.

    4. If value is a string:

      1. Append 0x0D 0x0A 0x0D 0x0A (CR LF CR LF) to chunk.

      2. Replace every occurrence of U+000D (CR) not followed by U+000A (LF), and every occurrence of U+000A (LF) not preceded by U+000D (CR), in value, by a string consisting of U+000D (CR) and U+000A (LF).

      3. Append the result of encoding value with encoding to chunk.

      4. Append 0x0D 0x0A (CR LF) to chunk.

      5. Append chunk to output chunks.

    5. Otherwise:

      1. Assert: value is a File.

      2. Append `; filename="`, followed by the result of escaping a multipart/form-data name given value’s name with encoding and isFilename set to true, followed by 0x22 0x0D 0x0A (" CR LF), to chunk.

      3. Let type be value’s type, if it is not the empty string, or "application/octet-stream" otherwise.

      4. Append `Content-Type: `, followed by the result of isomorphic encoding type, to chunk.

      5. Append 0x0D 0x0A 0x0D 0x0A (CR LF CR LF) to chunk.

      6. Append chunk, followed by value, followed by the byte sequence 0x0D 0x0A (CR LF), to output chunks.

  5. Append the byte sequence containing `--`, followed by boundary, followed by `--`, followed by 0x0D 0x0A (CR LF), to output chunks.

  6. Return the tuple boundary / output chunks.

This algorithm matches the behavior of all major browsers, except for the following bug:


The length of a multipart/form-data payload, given a list of chunks chunks which can be either byte sequences or Files, is the result of running the following steps:

  1. Let length be 0.

  2. For each chunk in chunks:

    1. If chunk is a byte sequence:

      1. Increase length by chunk’s length.

    2. Otherwise:

      1. Assert: chunk is a File.

      2. Increase length by chunk’s size.

  3. Return length.

To create a multipart/form-data readable stream from a list of chunks chunks which can be either byte sequences or Files, run the following steps:

  1. Let file stream be null.

  2. Let stream be a new ReadableStream.

  3. Let pull algorithm be an algorithm that runs the following steps:

    if file stream is null and chunks is not empty
    1. If chunks[0] is a byte sequence, enqueue a Uint8Array object wrapping an ArrayBuffer containing chunks[0] into stream.

    2. Otherwise:

      1. Assert: chunks[0] is a File object.

      2. Set file stream to the result of running chunks[0]'s stream method.

      3. Run pull algorithm.

    3. Remove the first item from chunks.

    if file stream is null and chunks is empty
    1. Close stream.

    if file stream is not null
    1. Let read request be a new read request with the following items:

      chunk steps, given chunk
      1. If chunk is not a Uint8Array object, error stream with a TypeError and abort these steps.

      2. Enqueue chunk into stream.

      close steps
      1. Set file stream to null.

      2. Run pull algorithm.

      error steps, given e
      1. Error stream with e.

    2. Let reader be the result of getting a reader for file stream.

    3. Read a chunk from reader with read request.

  4. Let cancel algorithm be an algorithm that runs the following steps, given reason:

    1. If file stream is not null, cancel file stream with reason.

  5. Set up stream with pullAlgorithm set to pull algorithm and cancelAlgorithm set to cancel algorithm.

  6. Return stream.

3. multipart/form-data parsing

These algorithms are a first attempt at defining a multipart/form-data parser for use in Body's formData() method. The current algorithms don’t yet match any browser because their behavior disagrees at various points.

Note that Gecko and Chromium also implement a Web Extensions API that parses multipart/form-data independently from the parser in Body (see Gecko bug 1697292):

chrome.webRequest.onBeforeRequest.addListener(
  (details) => {
    // Returns an object mapping names to an array of values represented by
    // either the string value or by the file’s filename.
    console.log(details.requestBody.formData);
  },
  {urls: ["<all_urls>"]},
  ["requestBody"]
);

The multipart/form-data parser takes a byte sequence input and a MIME type mimeType, and returns either an entry list or failure:

  1. Assert: mimeType’s essence is "multipart/form-data".

  2. If mimeType’s parameters["boundary"] does not exist, return failure. Otherwise, let boundary be the result of UTF-8 decoding mimeType’s parameters["boundary"].

    The definition of MIME type in [MIMESNIFF] has the parameter values being ASCII strings, but the parse a MIME type algorithm can create MIME type records containing non-ASCII parameter values. See whatwg/mimesniff issue #141. Gecko and WebKit accept non-ASCII boundary strings and then expect them UTF-8 encoded in the request body; Chromium rejects them instead.

  3. Let entry list be an empty entry list.

  4. Let position be a pointer to a byte in input, initially pointing at the first byte.

  5. While true:

    1. If position points to a sequence of bytes starting with 0x2D 0x2D (`--`) followed by boundary, advance position by 2 + the length of boundary. Otherwise, return failure.

    2. If position points to the sequence of bytes 0x2D 0x2D 0x0D 0x0A (`--` followed by CR LF) followed by the end of input, return entry list.

    3. If position does not point to a sequence of bytes starting with 0x0D 0x0A (CR LF), return failure.

    4. Advance position by 2. (This skips past the newline.)

    5. Let name, filename and contentType be the result of parsing multipart/form-data headers on input and position, if the result is not failure. Otherwise, return failure.

    6. Advance position by 2. (This skips past the empty line that marks the end of the headers.)

    7. Let body be the empty byte sequence.

    8. Body loop: While position is not past the end of input:

      1. Append the code point at position to body.

      2. If body ends with boundary:

        1. Remove the last 4 + (length of boundary) bytes from body.

        2. Decrease position by 4 + (length of boundary).

        3. Break out of body loop.

    9. If position does not point to a sequence of bytes starting with 0x0D 0x0A (CR LF), return failure. Otherwise, advance position by 2.

    10. If filename is not null:

      1. If contentType is null, set contentType to "text/plain".

      2. If contentType is not an ASCII string, set contentType to the empty string.

      3. Let value be a new File object with name filename, type contentType, and body body.

    11. Otherwise:

      1. Let value be the UTF-8 decoding without BOM of body.

    12. Add name / value to entry list.

To parse multipart/form-data headers, given a byte sequence input and a pointer into it position, run the following steps:

  1. Let name, filename and contentType be null.

  2. While true:

    1. If position points to a sequence of bytes starting with 0x0D 0x0A (CR LF):

      1. If name is null, return failure.

      2. Return name, filename and contentType.

    2. Let header name be the result of collecting a sequence of bytes that are not 0x0A (LF), 0x0D (CR) or 0x3A (:), given position.

    3. Remove any HTTP tab or space bytes from the start or end of header name.

    4. If header name does not match the field-name token production, return failure.

    5. If the byte at position is not 0x3A (:), return failure.

    6. Advance position by 1.

    7. Collect a sequence of bytes that are HTTP tab or space bytes given position. (Do nothing with those bytes.)

    8. Byte-lowercase header name and switch on the result:

      `content-disposition`
      1. Set name and filename to null.

      2. If position does not point to a sequence of bytes starting with `form-data; name="`, return failure.

      3. Advance position so it points at the byte after the next 0x22 (") byte (the one in the sequence of bytes matched above).

      4. Set name to the UTF-8 decoding without BOM of the result of collecting a sequence of bytes that are not 0x0A (LF), 0x0D (CR) or 0x22 ("), given position.

        The parsing of names and filenames is being worked on in issue #1.

      5. If the byte at position is not 0x22 ("), return failure. Otherwise, advance position by 1.

      6. If position points to a sequence of bytes starting with `; filename="`:

        1. Advance position so it points at the byte after the next 0x22 (") byte (the one in the sequence of bytes matched above).

        2. Set filename to the UTF-8 decoding without BOM of the result of collecting a sequence of bytes that are not 0x0A (LF), 0x0D (CR) or 0x22 ("), given position.

          The parsing of names and filenames is being worked on in issue #1.

        3. If the byte at position is not 0x22 ("), return failure. Otherwise, advance position by 1.

      `content-type`
      1. Let header value be the result of collecting a sequence of bytes that are not 0x0A (LF) or 0x0D (CR), given position.

      2. Remove any HTTP tab or space bytes from the end of header value.

      3. Set contentType to the isomorphic decoding of header value.

      Otherwise

      Collect a sequence of bytes that are not 0x0A (LF) or 0x0D (CR), given position. (Do nothing with those bytes.)

    9. If position does not point to a sequence of bytes starting with 0x0D 0x0A (CR LF), return failure. Otherwise, advance position by 2 (past the newline).

Intellectual property rights

Copyright © WHATWG (Apple, Google, Mozilla, Microsoft). This work is licensed under a Creative Commons Attribution 4.0 International License. To the extent portions of it are incorporated into source code, such portions in the source code are licensed under the BSD 3-Clause License instead.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[ENCODING]
Anne van Kesteren. Encoding Standard. Living Standard. URL: https://encoding.spec.whatwg.org/
[FETCH]
Anne van Kesteren. Fetch Standard. Living Standard. URL: https://fetch.spec.whatwg.org/
[FileAPI]
Marijn Kruisselbrink; Arun Ranganathan. File API. URL: https://w3c.github.io/FileAPI/
[HTML]
Anne van Kesteren; et al. HTML Standard. Living Standard. URL: https://html.spec.whatwg.org/multipage/
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[MIMESNIFF]
Gordon P. Hemsley. MIME Sniffing Standard. Living Standard. URL: https://mimesniff.spec.whatwg.org/
[STREAMS]
Adam Rice; Domenic Denicola; 吉野剛史 (Takeshi Yoshino). Streams Standard. Living Standard. URL: https://streams.spec.whatwg.org/
[URL]
Anne van Kesteren. URL Standard. Living Standard. URL: https://url.spec.whatwg.org/
[WebIDL]
Boris Zbarsky. Web IDL. URL: https://heycam.github.io/webidl/
[XHR]
Anne van Kesteren. XMLHttpRequest Standard. Living Standard. URL: https://xhr.spec.whatwg.org/

Informative References

[RFC2046]
N. Freed; N. Borenstein. Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types. November 1996. Draft Standard. URL: https://datatracker.ietf.org/doc/html/rfc2046
[RFC7578]
L. Masinter. Returning Values from Forms: multipart/form-data. July 2015. Proposed Standard. URL: https://datatracker.ietf.org/doc/html/rfc7578