UTF-8 character handling problem

[kerekesp]

Hi,

when I try to proxy a request with UTF-8 encoded body e.g. "á" , then the outgoing request length calculated incorrecty and the end of the body is truncated (by 2 x the number of UTF characters) and the output encoding will be wrong as well
"á" transformed to "á"
The proxy running on Tomcat, the default encoding is ISO-8859-1, most probable this is the problem. I tried every Tomcat config modificaton but the body handling has not been changed.
Can the proxy be changed to handle this e.g. adding the
servletRequest.setCharacterEncoding("UTF-8");
servletResponse.setCharacterEncoding("UTF-8");
based on the request header encoding parameter?

Thanks in advance

[dsmiley]

The proxy deals with bytes (io Streams), not characters (io Readers/Writers). The content length is byte oriented as well. So I'm suspicious there is anything wrong with the proxy.

[kerekesp]

Hi,
Thank you, you are right.

There was an extra logging in the newProxyRequestWithEntity where the input stream was read into a string, but not as UTF-8:

protected HttpRequest newProxyRequestWithEntity(String method, String proxyRequestUri, HttpServletRequest servletRequest) throws IOException {
BasicHttpEntityEnclosingRequest basicHttpEntityEnclosingRequest = new BasicHttpEntityEnclosingRequest(method, proxyRequestUri);
ServletInputStream servletInputStream = servletRequest.getInputStream();
StringBuilder sb = new StringBuilder();
int ch;
while ((ch = servletInputStream.read()) != -1)
sb.append((char)ch);
String body = sb.toString();
log.debug("Request body is: " + body);
basicHttpEntityEnclosingRequest.setEntity((HttpEntity)new InputStreamEntity(new ByteArrayInputStream(body.getBytes(StandardCharsets.UTF_8)), getContentLength(servletRequest)));
return (HttpRequest)basicHttpEntityEnclosingRequest;
}

Because we need to keep the loggging, so I've corrected the code:

protected HttpRequest newProxyRequestWithEntity(String method, String proxyRequestUri, HttpServletRequest servletRequest) throws IOException {
BasicHttpEntityEnclosingRequest basicHttpEntityEnclosingRequest = new BasicHttpEntityEnclosingRequest(method, proxyRequestUri);
BufferedReader reader = new BufferedReader(new InputStreamReader((InputStream)servletRequest.getInputStream(), StandardCharsets.UTF_8));
StringBuilder body = new StringBuilder();
char[] buffer = new char[1024];
for (int length = 0; (length = reader.read(buffer)) > 0;)
body.append(buffer, 0, length);
log.debug("Request body is: " + body);
basicHttpEntityEnclosingRequest.setEntity((HttpEntity)new InputStreamEntity(new ByteArrayInputStream(body.toString().getBytes(StandardCharsets.UTF_8)), getContentLength(servletRequest)));
return (HttpRequest)basicHttpEntityEnclosingRequest;
}

Thank you again, and sorry I'messed this modification in our code.
Regards,