Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect parsing of complex JS nested objects in POST request #400

Open
vrfdivino opened this issue Apr 7, 2022 · 5 comments
Open

Incorrect parsing of complex JS nested objects in POST request #400

vrfdivino opened this issue Apr 7, 2022 · 5 comments

Comments

@vrfdivino
Copy link

vrfdivino commented Apr 7, 2022

I created a simple Express server with one POST endpoint. The POST endpoint just return the request body sent from the client. The code for this server looks like this:

const express = require('express')

const app = express()
app.use(express.urlencoded({ extended: true}))
app.use(express.json())

app.post("/", (req,res) => {
        res.status(200).json({
                status: 'ok',
                data: req.body
        })
})

app.listen(8081, () => console.log('up and running on port 8081')

I also added a script file to test the said POST endpoint using needle. This is the code for the script file:

const needle = require('needle')

const postSomething = async () => {
        const res = await needle('post', 'http://localhost:8081', {
                field1: 'hello',
                field2: [
                        'line1',
                        'line2',
                        'line3',
                ],
                field3: [
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                {
                        subfield1: 'this is a string',
                        subfield2: ['this is array'],
                        subfield3: ['another array']
                },
                ]
        })
        console.log(res.body.data) // Note this line of code
}

postSomething()

Now, the output of console.log(res.body.data) does not match with the request body that was sent originally to the server. field3 becomes an array with single object instead of three and the subfield1 becomes an array. The output looks like this:

{
  field1: 'hello',
  field2: [ 'line1', 'line2', 'line3' ],
  field3: [ { subfield1: [Array], subfield2: [Array], subfield3: [Array] } ]
}

Now, to verify that the needle output is incorrect, I tested it in Postman. The correct output should look like this:

{
        "field1": "hello",
        "field2": [
            "line1",
            "line2",
            "line3"
        ],
        "field3": [
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            },
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            },
            {
                "subfield1": "this is a string",
                "subfield2": [
                    "this is array"
                ],
                "subfield3": [
                    "another array"
                ]
            }
        ]
    }

In summary, upon sending a complex nested objects in needle via POST request, it changes the structure of the original request body. To replicate the issue, this is my environment:
node --version v16.14.2
npm --version 8.5.0

{
  "name": "needle-json-post-error",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.3",
    "needle": "^3.0.0"
  }
}
@tomas
Copy link
Owner

tomas commented Apr 7, 2022

Are you sure the server is returning the JSON data in the correct format? Needle simply does a JSON.parse on the response data so I'm not sure if Needle's the culprit in this one.

@yetzt
Copy link
Contributor

yetzt commented Apr 7, 2022

looks to me like this happens when the post data is sent in querystring format (default). use { json: true } in your needle options to send the data as json.

i would refrain from using querystring format to transfer complex objects, because to my knowledge there is no consistant standard about how to encode and decode complex objects. see for yourself:

const qs = require("querystring");

const data = {
	field1: 'hello',
	field2: [
		'line1',
		'line2',
		'line3',
	],
	field3: [
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	{
		subfield1: 'this is a string',
		subfield2: ['this is array'],
		subfield3: ['another array']
	},
	]
};

console.log(qs.decode(qs.encode(data)));

needle uses this method to encode the data, and the express.urlencoded middleware that ultimately parses the data comes from here.

@tomas
Copy link
Owner

tomas commented Apr 7, 2022

Oh I see. I misunderstood then; I thought the problem was related the the decoding of JSON, not the encoding part. Ok I'll take a look in a while.

@tomas
Copy link
Owner

tomas commented Apr 7, 2022

Ok, we can try and see whether adding the array item index this line makes it works like expected. The resulting encoded string would look like this (using your example data):

image

@yetzt
Copy link
Contributor

yetzt commented Apr 7, 2022

maybe writing a test utilizing the body-parser middleware from express would be in order. keep in mind that this would fix it for one kind of backend, other backends like php might end up not parsing it in the same way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants