Special characters are wrongly encoded/decoded

[vpas88]

I have a small application using falcon-api, URI-s can have special characters like:

íéáűúőóüö

When I send above URI, it's percent-encoded to:

%C3%AD%C3%A9%C3%A1%C5%B1%C3%BA%C5%91%C3%B3%C3%BC%C3%B6

I use gunicorn, gunicorn pass this to falcon as:

íéáÃ

After it's encoded+decoded in request.py here (I guess):

if not isascii(path):
            path = path.encode('iso-8859-1').decode('utf-8', 'replace')

The result is:

±ÃºÃ

Instead of the original:

íéáűúőóüö

[vytas7]

Hi @vpas88!
I've converted this issue to a discussion since I cannot really reproduce any of the above. By the referenced line, Falcon is merely following the PEP 3333 convention to decode tunneled bytes from Latin-1.

[vytas7]

I've used the following test.py:

import falcon


class Inspector:
    def on_get(self, req, resp, param):
        resp.media = {
            'headers': req.headers,
            'method': req.method,
            'param': param,
            'uri': req.uri,
        }


app = falcon.App()
app.add_route('/{param}', Inspector())

Packages used (on CPython 3.10):

$ pip freeze | egrep '(falcon|gunicorn)'
falcon==3.1.1
gunicorn==21.2.0

Sending a request similar to yours:

$ xh http://localhost:8000/íéáűúőóüö
HTTP/1.1 200 OK
Connection: close
Content-Length: 250
Content-Type: application/json
Server: gunicorn

{
    "headers": {
        "ACCEPT-ENCODING": "gzip, deflate, br",
        "USER-AGENT": "xh/0.18.0",
        "CONNECTION": "keep-alive",
        "ACCEPT": "*/*",
        "HOST": "localhost:8000"
    },
    "method": "GET",
    "param": "íéáűúőóüö",
    "uri": "http://localhost:8000/íéáűúőóüö"
}

Could it be that you had a very old version of Gunicorn, Falcon, or just your HTTP client encodes or decodes not in a way you would expect it? Which client or browser are you using?

[vpas88]

I have executed the following with your code (i used curl) and I see the same problem:

(venv) vpas@alderaan1:~/dev/test$ cat test.py
import falcon


class Inspector:
    def on_get(self, req, resp, param):
        resp.media = {
            'headers': req.headers,
            'method': req.method,
            'param': param,
            'uri': req.uri,
        }


app = falcon.App()
app.add_route('/{param}', Inspector())
(venv) vpas@alderaan1:~/dev/test$ gunicorn test:app test.py 
[2023-09-17 20:14:43 +0200] [1077018] [INFO] Starting gunicorn 21.2.0
[2023-09-17 20:14:43 +0200] [1077018] [INFO] Listening at: http://127.0.0.1:8000 (1077018)
[2023-09-17 20:14:43 +0200] [1077018] [INFO] Using worker: sync
[2023-09-17 20:14:43 +0200] [1077019] [INFO] Booting worker with pid: 1077019
^Z
[1]+  Stopped                 gunicorn test:app test.py
(venv) vpas@alderaan1:~/dev/test$ bg
[1]+ gunicorn test:app test.py &
(venv) vpas@alderaan1:~/dev/test$ curl http://localhost:8000/íéáűúőóüö
{"headers": {"HOST": "localhost:8000", "USER-AGENT": "curl/7.81.0", "ACCEPT": "*/*"}, "method": "GET", "param": "íéáűúÅóüö", "uri": "http://localhost:8000/íéáűúÅóüö"}
(venv) vpas@alderaan1:~/dev/test$ pip freeze | egrep '(falcon)|gunicorn'
falcon==3.1.1
falcon-apispec==0.4.0
falcon-auth==1.1.0
gunicorn==21.2.0

As you can see I have the same versions like you.

[vpas88]

OK, I think it's not react/axios. My setup is the following (in detail):
React Client -> Nginx -> Gunicorn -> Falcon

Nginx is still logging the correct percent-encoded URI, however Gunicorn is already passed "wrong" character to Falcon request.py.
So now I'm suspecting Nginx is doing some unneeded percent-decoding...

[vytas7]

Don't think Nginx is at fault, I think it normally rejects non-ASCII URLs with a 400 🤔

[vpas88]

I have created a little socket server to see what NGINX send to gunicorn - now my gunicorn was that:

#! /usr/bin/python
# a simple tcp server
import socket,os
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 19960))
sock.listen()
while True:
    try:
        connection,address = sock.accept()
        buf = connection.recv(1024)
        print(buf.decode())
    except KeyboardInterrupt:
        connection.close()
        break
    connection.close()

Result (actually it's a DELETE request, but i dont think it's important):

DELETE /íéáűúőóüö HTTP/1.0
Host: 127.0.0.1:19960
Connection: close
sec-ch-ua: "Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"
accept: application/json, text/plain, */*
sec-ch-ua-mobile: ?0

Chrome sent this (the answer was not properly handled, but you can see that it's sent percentage-encoded URI):

So my understanding is that percentage-encoded URI is properly sent from client side (React->axios->Chrome) towards NGINX where it's decoded and the utf-8(?) string sent towards gunicorn what is somehow translates this to wrong and this is sent to falcon.

[vpas88]

OK, finally I have manage to solve the problem. I'm putting the solution here for future reference; when using NGINX and URI is parsed in location proxy_pass will get the decoded value.
My solution was that:

location ~ ^/api/(.*)$ {
	rewrite  ^/api/(.*)$ $1$is_args$args break;
	proxy_pass http://127.0.0.1:19960;
}

This way the original percentage-encoded format is passed towards gunicorn/falcon and they will parse it properly.

[vytas7]

Good that you found it 👍

I've never had this specific issue, but had other edge cases, so in general I try not to design APIs that pass arbitrary values inside URI.

[CaselIT]

I've changed this discussion in a Q&A and marked that as the answer. thanks for the update