Reconfiguring built-in Prometheus listeners (TCP ports 20201, 20202)

[gopikaratheesh]

NOTE: To get the best support experience for bug fixes, please go to https://cloud.google.com/support-hub and follow the instructions. In comparison, Bug reports filed in this repo only have best effort support, and do not have guaranteed response / resolution SLOs

Describe the bug
Prometheus exporter TCP ports (20201, 20202) are enabled by default on Ops Agent, which causes problems for users wanting to bind to those ports for other purposes, or reduce network exposure.

While the Prometheus listeners are fairly minimal (a fairly simple handler for /metrics), since the daemons run as root, users wanting to run Ops Agent in a security–sensitive environment will want to eliminate inbound requests.

Additionally, users wanting to run their own service binding to TCP ports 20201 or 20202 will run into conflicts.

There's no obvious way to reconfigure these ports, whether changing the binding address or port numbers. At the very least, it should be possible to bind these ports to localhost instead (::1 or 127.0.0.1).

I recognise that these ports are used by Ops Agent for self monitoring, so avoiding listening on the ports entirely is likely infeasible.

To Reproduce
Steps to reproduce the behavior:

Environment: "Red Hat Enterprise Linux 8.10 (Ootpa)"
Use default config
Run netstat -anp | grep :2020:

tcp6       0      0 :::20201                :::*                    LISTEN      15072/otelopscol
tcp        0      0 0.0.0.0:20202           0.0.0.0:*               LISTEN      15123/fluent-bit
tcp        0      0 127.0.0.1:20202         127.0.0.1:45340         ESTABLISHED 15123/fluent-bit
tcp        0      0 127.0.0.1:45340         127.0.0.1:20202         ESTABLISHED 15072/otelopscol
tcp        0      0 127.0.0.1:51388         127.0.0.1:20201         ESTABLISHED 15072/otelopscol
tcp6       0      0 127.0.0.1:20201         127.0.0.1:51388         ESTABLISHED 15072/otelopscol

Observe this config in /run/google-cloud-ops-agent-opentelemetry-collector/otel.yaml:

telemetry:
metrics:
address: 0.0.0.0:20201

Observe this config in /run/google-cloud-ops-agent-fluent-bit/fluent_bit_main.conf:

[OUTPUT]
Match *
Name prometheus_exporter
host 0.0.0.0
port 20202

Expected behavior
I would expect to be able to reconfigure to have fluent-bit run in user-specified ports

Environment (please complete the following information):

• VM distro / OS: Red Hat Enterprise Linux 8.10 (Ootpa)
• Ops Agent version [e.g. 2.14.0] : google-cloud-ops-agent-2.52.0-1.el8.x86_64
• Ops Agent configuration: default config