K6 Does not respect the boundary value placed in a Content-Type header

[Ardesco]

If you specify a Context-Type header for a multipart form with a specific boundary, it is ignored:

let parameters = {
        headers: {
            "Authorization": "someAuthToken",
            "Content-Type": "multipart/form-data; boundary=--myBoundary",
            "Accept-Encoding": "br, gzip, deflate"
        }
    };

When looking at the logged request, you get:

POST /some/url HTTP/1.1
Host: api.feat1.dev.imaginecurve.com
User-Agent: k6/0.22.1 (https://k6.io/);
Content-Length: 507
Accept-Encoding: br, gzip, deflate
Authorization: someAuthToken
Content-Type: multipart/form-data; boundary=--myBoundary
Cookie: device_view=full

--87154a0502b4f0b523bb4fc2571b3d87d741f6e7ca73baaad7f8ab5eb74a
Content-Disposition: form-data; name="foo"

foo content
--87154a0502b4f0b523bb4fc2571b3d87d741f6e7ca73baaad7f8ab5eb74a
Content-Disposition: form-data; name="bar"

bar content
--87154a0502b4f0b523bb4fc2571b3d87d741f6e7ca73baaad7f8ab5eb74a
Content-Disposition: form-data; name="_dummyFile"; filename="placeholder"
Content-Type: application/octet-stream


--87154a0502b4f0b523bb4fc2571b3d87d741f6e7ca73baaad7f8ab5eb74a--

Note that the boundary separator is not the one specified in the Content-Type. This results in clients being unable to parse the payload because it can't find the specified boundary.

I would suggest this is linked to #747

[Ardesco]

The workaround is to just not specify a Content-Type, K6 then automatically generates one with the correct boundary definition.

[na--]

Thanks, you're right, we should fix this at the same time we fix #747, since both issues are very connected!

[IsabelaPastorini]

In the case that I MUST change manually the 'Content-type' and then add a random boundary value doesn't work. My scenario is:

• I receive from a flowContext my header params and the content-type is set automatically to 'application/json'
• Since I'm testing an endpoint that requires a 'multipart/form-data' I have to manually change the header content-type and added a random boundary value.

let binPdf = open("../bin-pdf.pdf", "b");
param.headers['Content-Type'] = "multipart/form-data;boundary=random-boundary-value";
        var data = {
            file: http.file(binPdf, "bin-pdf.pdf", "application/pdf") 
        };
let res = http.post(url, data, params);

K6 still complaining of Missing initial multi part boundary

[imiric]

@IsabelaPastorini Could you try this workaround suggested in #1571?

Unfortunately you might need to build the entire body manually, or create it outside of k6 and load the binary that way.

This issue is in our backlog, but I can't say an ETA for a fix, as other issues currently have higher priority. Sorry for the inconvenience!

[IsabelaPastorini]

It worked fine! Unfortunately pdf file converted to base64 is to big, so I setUp random data like "123" to convert and worked, Thanks or the suggestion.
Since k6 pick the correct boundary-value from your file and return a header with this value, can we check if the header is present in the request and change it to the correct one? I take a look in the request.go, more specifically in
handleObjectBody := func(data map[string]interface{})

result := &parsedHTTPRequest{
--
  | url: &reqURL,
  | req: &http.Request{
  | Method: method,
  | URL:    reqURL.URL,
  | Header: make(http.Header),
  | },


result.req.Header.Set("Content-Type", mpw.FormDataContentType())

Creates a new Header and then fill Contet-type with the correct value.

[imiric]

Hey guys, just a heads-up: multipart requests should be simpler to make now with the FormData polyfill and k6 v0.30.0 (make sure you're using at least this version). See the documentation.

[IsabelaPastorini]

For pdf i still using the #1571 . Can you suggest how can I use Form Data to deal with pdf? I read a little about it and converting the file to binary still needed (?), other wise I receive the error:

GoError: unknown request body type goja.ArrayBuffer

const pdf = open('/path/to/pdf-file.pdf');
export default function() {
  const fd = new FormData();
  fd.append('pdf-file', http.file(pdf, 'pdf-file.pdf', 'application/pdf'));
  const res = http.post('https://httpbin.test.k6.io/post', fd.body(),
    { headers: { 'Content-Type': 'multipart/form-data; boundary=' + fd.boundary }});
  check(res, {
    'is status 200': (r) => r.status === 200,
  });
}

[mstoykov]

Hi @IsabelaPastorini ,

this is supported only since k6 v0.30.0, and from the error message, it seems you are using an earlier version.

Can you confirm that and if not, upgrade and retry?

[IsabelaPastorini]

I will try

[IsabelaPastorini]

Still doesn't working... My approach is correct?

[imiric]

@IsabelaPastorini Make sure you're opening the file as binary (with the 'b' argument). See the example and again make sure you're using k6 v0.30.0.

Otherwise let us know what's not working... Any errors? It's not the most efficient polyfill, so that can probably be improved, but it works, I just tested it with a PDF.

[IsabelaPastorini]

Below there is my code, i've checked more than once more that the k6 version is v0.30.0, the bug occurs when I do the http.post. I can read the fd.boundary correctly. If there's any mistake in my approach please let me know, I've followed the https://k6.io/docs/examples/data-uploads#advanced-multipart-request

const pdf_file = open('test.pdf','b');

export default function() {
       const fd = new FormData();
       fd.append('pdf-file', http.file(pdf_file, 'test.pdf', 'application/pdf'));
        let url = 'https://httpbin.test.k6.io/post';
        let res = http.post(url, fd.body(),
        { headers: { 'Content-Type': 'multipart/form-data; boundary=' + fd.boundary }});
      check(res, {
        'is status 200': (r) => r.status === 200,
      });
}

[imiric]

the bug occurs when I do the http.post

What bug? Again, let us know the exact error you're getting, otherwise we can't help you. :)

I tested with the same example you posted, and there were no errors. Here's the full runnable script:

import http from 'k6/http';
import { check } from 'k6';
import { FormData } from 'https://jslib.k6.io/formdata/0.0.1/index.js';

const pdf_file = open('test.pdf','b');

export default function() {
  const fd = new FormData();
  fd.append('pdf-file', http.file(pdf_file, 'test.pdf', 'application/pdf'));
  let url = 'https://httpbin.test.k6.io/post';
  let res = http.post(url, fd.body(),
    { headers: { 'Content-Type': 'multipart/form-data; boundary=' + fd.boundary }});
  check(res, {
    'is status 200': (r) => r.status === 200,
  });
}

Running k6 run --http-debug=full test.js outputs:

INFO[0000] Request:
POST /post HTTP/1.1
Host: httpbin.test.k6.io
User-Agent: k6/0.30.0 (https://k6.io/)
Content-Length: 3227
Content-Type: multipart/form-data; boundary=------RWWorkerFormDataBoundary0.qv0h1c1dwy
Accept-Encoding: gzip

--------RWWorkerFormDataBoundary0.qv0h1c1dwy
Content-Disposition: form-data; name="pdf-file"; filename="test.pdf"
Content-Type: application/pdf

%PDF-1.3
[...]

--------RWWorkerFormDataBoundary0.qv0h1c1dwy--
  group= iter=0 request_id=2814822f-2baf-47da-68e2-fef7bcf3d8bb scenario=default source=http-debug vu=1

INFO[0001] Response:
HTTP/2.0 200 OK
Content-Length: 4545
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Fri, 19 Feb 2021 09:38:33 GMT
Server: gunicorn/19.9.0

{
  "args": {},
  "data": "",
  "files": {
    "pdf-file": "data:application/pdf;base64,..."
  },
  "form": {},
  "headers": {
    "Content-Length": "3227",
    "Content-Type": "multipart/form-data; boundary=------RWWorkerFormDataBoundary0.qv0h1c1dwy",
    "Host": "httpbin.test.k6.io",
    "User-Agent": "k6/0.30.0 (https://k6.io/)",
    "X-Amzn-Trace-Id": "Root=1-602f8719-3d1dff6a45043c100b5a2396"
  },
  "json": null,
  "origin": "...",
  "url": "https://httpbin.test.k6.io/post"
}
  group= iter=0 request_id=2814822f-2baf-47da-68e2-fef7bcf3d8bb scenario=default source=http-debug vu=1

I snipped the binary and base64 data, but be careful with --http-debug=full when dealing with binary files as it might mess up your terminal, so I tested with a small sample PDF file.

[imiric]

The FormData polyfill resolves this issue, and is the recommended way to submit multipart requests, so I'll close this issue.

In the future, we might integrate FormData into k6 as part of the new HTTP API (initial design document), but we haven't decided on the details yet.