apache image logging client ip address behind reverse proxy

[liejuntao001]

Problem:
I'm running docker image
nextcloud:18.0.4-apache
behind a reverse proxy (traefik)

I've setup like this:
the reverse proxy provides correct header "X-Real-IP"
apache2 remoteip is effective
nextcloud.log shows correct client ip address
apache2 log does not show correct client ip address, instead it shows the ip of the reverse proxy

Fix recommendation:
Modify /etc/apache2/apache2.conf to replace the %h with %a in LogFormat, or an equivalent place to override.

LogFormat "%v:%p %a %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i"" vhost_combined
LogFormat "%a %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i"" combined
LogFormat "%a %l %u %t "%r" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Fix references:
docker-library/wordpress#383 (comment)
https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html

Commit reference:
docker-library/wordpress@a4eef48

[kissi7]

Hello
changing %h with %a in LogFormat did not work for me. (I hope my trusted proxy is covered by 10.0.0.0/8 so I changed it to %{X-Forwarded-For} and it works fine.

[jlegido]

@kissi7 I would really appreciate if you share your working /etc/apache2/apache2.conf, many thanks in advance.

[jlegido]

@kissi7 disregard, snippet of my working config:

LogFormat "%v:%p %h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

[HJunyuan]

Changing the /etc/apache2/apache2.conf to the above example only helps to display the correct IP in the logs.

However, the Nextcloud app still does not seem to be obtaining the correct IP from the Nginx reverse proxy that was supplied in the examples folder.

I was able to verify this by installing the Restrict login to IP addresses app to try and restrict specific external IPs. However, it does not work as expected. Restricting any external IP addresses will not block their logins. If I were to restrict internal Docker IPs like 172.19.0.x, all logins become blocked - proving that Nextcloud is not obtaining IPs from the Nginx reverse proxy headers.

Is anyone able to help with getting Nextcloud to obtain the correct IPs from the Nginx reverse proxy? I've followed all the instructions (i.e. adding trusted_proxies in config.php, etc.) but nothing seems to work.

[jlegido]

@HJunyuan the key concept here is the ability of the reverse proxy (I have tested it with nginx but no luck yet with traefik) to add some headers to the request. In my previous example I'm using X-Forwarded-For header.

I'm afraid you use case it outside the topic of this issue, since you are talking about a plugin which I have no idea about it.

Question: are you able to log the header X-Forwarded-For ?

[HJunyuan]

@jlegido no worries at all, I think I will raise a new issue on this.

As I saw this thread referenced in #570, I just wanted to point out that while the logs will show the correct IP, there is still an issue where the Nextcloud app is not using the correct IP from the header. The plugin I mention merely serves as an example of a case affected by this issue.

So to answer your question: Yes, I was able to log the X-Forwarded-For header in the logs.

[jlegido]

@HJunyuan OK, so we narrow down the issue to the plugin, and how it get the public IP address of the request in order to allow or deny it.

I agree that makes sense to open a separate issue to track it, good luck

[tetebueno]

Hi, I'm experiencing something similar. I get the wrong IP on the Apache logs (the docker logs output) but the application (I mean, the nextcloud.log entries) and the Geoblocker plugin both get the right IP.

What's wrong with the %h to %a in apache.conf change?

[tetebueno]

My bad. For some reason APACHE_DISABLE_REWRITE_IP was set to 1 when starting the container. I removed that assignment and now I get real IPs in the Apache logs.

Also, I had tu use a custom Dockerfile like this:

FROM nextcloud:20.0.4-apache

RUN sed -E -i 's/(LogFormat.+)%h(.+)/\1%a\2/g' /etc/apache2/apache2.conf

So sed would set the appropriate format to the log entries.

Hope it helps.

[ElusiveMind]

If you are going to use %a in your LogFormat directive, be sure mod_remoteip is enabled and you have

# Define our proxy through IP header (requires mod_remoteip)
RemoteIPHeader X-Forwarded-For

In your apache configuration. This does not include any other proxying you might need to configure up-stream.

[tetebueno]

For the record, as it's seen here, remoteip module is enabled by default in the Apache image. Also (you can see that in the same line), X-Real-IP is used by default instead of X-Forwarded-For if you don't want to mess with any extra configuration.

[Samhamsam]

I have the latest docker image from nextcloud (22.1.0-apache), and still i get the nginx proxy ip address on docker logs. The reverse proxy is an nginxproxy/nginx-proxy container. No idea why it does show 172.20.0.3...

172.20.0.3 - - [20/Aug/2021:19:53:10 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 251 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

[jlegido]

I have the latest docker image from nextcloud (22.1.0-apache), and still i get the nginx proxy ip address on docker logs. The reverse proxy is an nginxproxy/nginx-proxy container. No idea why it does show 172.20.0.3...

172.20.0.3 - - [20/Aug/2021:19:53:10 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 251 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"

@Samhamsam Kindly:

1. Re-deploy a fresh instance

2. Apply proposed config

3. Restart docker container

4. Tail nextcloud docker container logs to see if logging public IP address

If above solution failed:

1. Deploy Whoami docker image

2. Configure your reverse proxy to send traffic to this backend

3. Check 2 things:

a) The reverse proxy is sending X-Forwarded-For (I assume yes)

b) X-Forwarded-For header is containing public IP address of client. This step is important, because depending on your network infrastructure maybe you are always getting a private IP address (in my case I was behind a VPN traversing traefik)

[riyad]

FWIW: just changing %h to %a doesn't do the trick in all cases. I just found out that clients coming from private IP ranges were still ignored (i.e. only the proxy was logged). Only after changing the RemoteIPTrustedProxy directives to RemoteIPInternalProxy the correct client IPs appeared in the Apache logs.

[Adito5393]

@riyad Thanks a lot for your comment!
For people using the Cloudflare proxy feature, just overwrite the /etc/apache2/conf-available/remoteip.conf file with:

# Cloudflare IPs: https://www.cloudflare.com/ips/
RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 10.0.0.0/8
RemoteIPTrustedProxy 172.16.0.0/12
RemoteIPTrustedProxy 192.168.0.0/16
RemoteIPTrustedProxy 173.245.48.0/20
RemoteIPTrustedProxy 103.21.244.0/22
RemoteIPTrustedProxy 103.22.200.0/22
RemoteIPTrustedProxy 103.31.4.0/22
RemoteIPTrustedProxy 141.101.64.0/18
RemoteIPTrustedProxy 108.162.192.0/18
RemoteIPTrustedProxy 190.93.240.0/20
RemoteIPTrustedProxy 188.114.96.0/20
RemoteIPTrustedProxy 197.234.240.0/22
RemoteIPTrustedProxy 198.41.128.0/17
RemoteIPTrustedProxy 162.158.0.0/15
RemoteIPTrustedProxy 104.16.0.0/13
RemoteIPTrustedProxy 104.24.0.0/14
RemoteIPTrustedProxy 172.64.0.0/13
RemoteIPTrustedProxy 131.0.72.0/22

And reload the apache within the docker container: /etc/init.d/apache2 reload

To make the change persistent after a docker restart, bind the file from the host or use the COPY cmd inside the Dockerfile.

[isdnfan]

According to documentation expected behavior is to accept proxy X-FORWARDED-FOR headers from RFC 1918 private IP address ranges. Internally Apache mod_remoteip is used to achieve this. mod_remoteip knows 2 different trusted proxy directives RemoteIPTrustedProxy and RemoteIPInternalProxy.

from mod_remoteip

The RemoteIPInternalProxy directive adds one or more addresses (or address blocks) to trust as presenting a valid RemoteIPHeader value of the useragent IP. Unlike the RemoteIPTrustedProxy directive, any IP address presented in this header, including private intranet addresses, are trusted when passed from these proxies.

This image defines 3 RFC 1198 ip ranges as RemoteIPTrustedProxy inside /etc/apache2/conf-available/remoteip.conf which makes the config useless for almost every installation - as installations using private addressing for reverse proxy usually use private addresses for clients as well. but private addresses (both IPv4 and IPv6) are ignored by RemoteIPTrustedProxy directive which requires APACHE_DISABLE_REWRITE_IP and TRUSTED_PROXIES for such installations and only connections from the internet are logged with real client IP.

Best solution would be to replace RemoteIPTrustedProxy with RemoteIPInternalProxy in /etc/apache2/conf-available/remoteip.conf as previously suggested in #1068. This solution address both Nextcloud logging and Apache logging. For advanced scenario like Cloudflare I would recommend to mount remoteip.conf into the docker host as it's hard or even impossible to configure such using environment variables. This would allow to remove APACHE_DISABLE_REWRITE_IP (and potentially TRUSTED_PROXIES as well).

#1426 #1494

[egabosh]

Moin,

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8
RemoteIPInternalProxy 172.16.0.0/12
RemoteIPInternalProxy 192.168.0.0/16

in remoteip.conf works for me with a Traefik ReverseProxy

But what about ipv6 addresses?

There again only the local IPv6 IP of the proxy is shown to me in the Docker log:
fdaa:a192:b168:cd41::1 - - [31/May/2023:11:36:40 +0200] "GET /login HTTP/1.1" 200 16360 "-" "Wget/1.21"

I already tried
RemoteIPTrustedProxy fdaa:a192:b168:cd41::1
as well as
RemoteIPInternalProxy fdaa:a192:b168:cd41::1
without success.

Any ideas?

[isdnfan]

please try this remoteip.conf

# RFC1918
RemoteIPInternalProxy 10.0.0.0/8
RemoteIPInternalProxy 172.16.0.0/12
RemoteIPInternalProxy 192.168.0.0/16
# unique local addresses 
# fc00::/7 is divided into 2 different /8 networks  fc00::/8 and fd00::/8
RemoteIPInternalProxy fc00::/7
# Link-local addresses have a prefix of FE80::/10
RemoteIPInternalProxy fe80::/10
# reserved for documentation, widely used in tutorials.
RemoteIPInternalProxy 2001:db8::/32

[egabosh]

@isdnfan thanks for the quick reply but unfortunately it did not help

[egabosh]

OK, found a solution for ipv6.
In the /etc/docker/daemon.json must be set among others "ip6tables": true. Here is an example:

{
  "ipv6": true,
  "fixed-cidr-v6": "fdab::/64",
  "experimental": true, 
  "ip6tables": true
}

After changing don't forget to restart the docker-daemon, e.g.:
systemctl restart docker.service

Looks like no change (RemoteIPInternalProxy) is needed to apache config.

I am a little unsure and hope that this configuration has no disadvantages, especially with regard to stability and security.

[UltraBlackLinux]

This is still not fixed for me. I just pulled the "latest" image again, and I'm still getting this behavior, making the brute force protection unusable. All I did notice is that the IP in the PR differs from the IP that I see in the logs: 172.16.0.0 vs 172.18.0.2 I have no idea about apache, but that might be one reason why it's not working

[joshtrichards]

@UltraBlackLinux As the original issue has been fixed, and your situation is occurring apparently after #2004, please create a dedicated issue with your associated configs.