Jv two processes listening on the same port

[JoukoVirtanen]

Description

Adds an integration test that starts two processes which listen on the same port. There have been reports of duplicate listening endpoints. This PR is part of the investigation into that issue.

Checklist

• Investigated and inspected CI test results
  - [ ] Updated documentation accordingly

Automated testing
- [ ] Added unit tests

• Added integration tests
  - [ ] Added regression tests

If any of these don't apply, please comment below.

Integration tests don't need unit or regression tests.

Testing Performed

Running the integration test locally

$ sudo make -C integration-tests TestReuseport COLLECTOR_TAG=3.16.x-82-gf1e4d3ca19
make: Entering directory '/home/jvirtane/projects/collector/integration-tests'
docker rm -fv container-stats benchmark collector grpc-server 2>/dev/null || true
go version
go version go1.20.4 linux/amd64
set -o pipefail; go test -tags bench -timeout 60m -count=1 -v -run TestReuseport 2>&1 | tee -a integration-test.log
=== RUN   TestReuseport
=== RUN   TestReuseport/TestReuseport
{Name:sh ExePath:/bin/sh Uid:0 Gid:0 Pid:274978 Args:-c python3 so_reuseport.py & python3 so_reuseport.py meaningless_argument} 
{Name:python3 ExePath:/usr/bin/python3 Uid:0 Gid:0 Pid:275055 Args:so_reuseport.py} 
{Name:python3 ExePath:/usr/bin/python3 Uid:0 Gid:0 Pid:275056 Args:so_reuseport.py meaningless_argument} 



{Protocol:L4_PROTOCOL_TCP Address:0xc0001c9b90 CloseTimestamp:(timestamp: nil Timestamp) Originator:0xc0001c9b00} 
{ProcessName:python3 ProcessExecFilePath:/usr/bin/python3 ProcessArgs:so_reuseport.py} 
{Protocol:L4_PROTOCOL_TCP Address:0xc0001c9d10 CloseTimestamp:(timestamp: nil Timestamp) Originator:0xc0001c9c80} 
{ProcessName:python3 ProcessExecFilePath:/usr/bin/python3 ProcessArgs:so_reuseport.py meaningless_argument} 
CPU: Container grpc-server, Mean 0.006842105263157895, StdDev 0.02749800630731604
CPU: Container collector, Mean 0.898421052631579, StdDev 1.061671968369257
Writing perf.json
--- PASS: TestReuseport (76.36s)
    --- PASS: TestReuseport/TestReuseport (0.00s)
PASS
ok  	github.com/stackrox/collector/integration-tests	76.364s
make: Leaving directory '/home/jvirtane/projects/collector/integration-tests'

Running the process and checking with lsof and netstat

[jvirtane@jvirtane so_reuseport]$ python3 so_reuseport.py & python3 so_reuseport.py meaningless_argument &
[1] 339720
[2] 339721
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ lsof -i -n -P | grep 8080
python3   339720 jvirtane    3u  IPv4 1179939      0t0  TCP *:8080 (LISTEN)
python3   339721 jvirtane    3u  IPv4 1171211      0t0  TCP *:8080 (LISTEN)
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ netstat -tulpn | grep 8080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      339720/python3  

Interestingly lsof reports two listening endpoints, while netstat reports one.

Checking namespace inodes

[jvirtane@jvirtane so_reuseport]$ docker container ls
CONTAINER ID   IMAGE                                                       COMMAND                  CREATED          STATUS          PORTS     NAMES
02d13934c40f   babcb01bc189                                                "/bin/sh -c 'python3…"   9 seconds ago    Up 8 seconds              competent_hypatia
2dabe805269d   quay.io/stackrox-io/collector:3.16.x-90-g13fadebc8f-dirty   "/bootstrap.sh /bin/…"   51 minutes ago   Up 51 minutes             collector
909131df2f09   quay.io/rhacs-eng/grpc-server:4.1.x-780-g5d8b38fe4d         "/mock-grpc-server"      51 minutes ago   Up 51 minutes             grpc-server
176a71923540   quay.io/rhacs-eng/collector-performance:stats-1.0.1         "/stats.sh"              51 minutes ago   Up 51 minutes             container-stats
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ 
[jvirtane@jvirtane so_reuseport]$ docker exec -it 02d13934c40f /bin/bash
root@02d13934c40f:/# 
root@02d13934c40f:/# 
root@02d13934c40f:/# cd /proc
root@02d13934c40f:/proc# ls
1   9          bus       cpuinfo    dma          filesystems  ioports   key-users    kpagecount  mdstat   mounts        partitions   self      swaps          thread-self  version
17  acpi       cgroups   crypto     driver       fs           irq       keys         kpageflags  meminfo  mtrr          sched_debug  slabinfo  sys            timer_list   vmallocinfo
7   asound     cmdline   devices    execdomains  interrupts   kallsyms  kmsg         loadavg     misc     net           schedstat    softirqs  sysrq-trigger  tty          vmstat
8   buddyinfo  consoles  diskstats  fb           iomem        kcore     kpagecgroup  locks       modules  pagetypeinfo  scsi         stat      sysvipc        uptime       zoneinfo
root@02d13934c40f:/proc# cat 7/comm
python3
root@02d13934c40f:/proc# 
root@02d13934c40f:/proc# 
root@02d13934c40f:/proc# cat 8/comm
python3
root@02d13934c40f:/proc# ls -l 7/fd
total 0
lr-x------. 1 root root 64 Sep 19 05:36 0 -> /dev/null
l-wx------. 1 root root 64 Sep 19 05:36 1 -> 'pipe:[177909]'
l-wx------. 1 root root 64 Sep 19 05:36 2 -> 'pipe:[177910]'
lrwx------. 1 root root 64 Sep 19 05:36 3 -> 'socket:[185442]'
root@02d13934c40f:/proc# 
root@02d13934c40f:/proc# 
root@02d13934c40f:/proc# ls -l 8/fd
total 0
lrwx------. 1 root root 64 Sep 19 05:36 0 -> /dev/null
l-wx------. 1 root root 64 Sep 19 05:36 1 -> 'pipe:[177909]'
l-wx------. 1 root root 64 Sep 19 05:36 2 -> 'pipe:[177910]'
lrwx------. 1 root root 64 Sep 19 05:36 3 -> 'socket:[183618]'

The inodes can be found from the entries with socket:[<inode>]. Here the inodes are different. Collector only reports listening endpoints with unique container ID and network namespace inodes. This is why it can report both listening endpoints.

Related PR

Note that this PR is similar to #1329

The difference is that in that PR the two processes created have a parent child relationship and here they do not.

[JoukoVirtanen]

Note that without meaningless_argument only one listening endpoint is reported. Perhaps there should also be a test for this case.

[github-actions]

Kernel	Method	Without Collector Time (secs)	With Collector Time (secs)	Baseline median (secs)	Collector median (secs)	PValue