Unexpected Exit Code 1 on SIGTERM Signal Handling in nginx-prometheus-exporter

[knziiy]

Description

I've encountered an issue with the nginx-prometheus-exporter where it returns an exit code of 1 when handling the SIGTERM signal, despite explicitly calling os.Exit(0) in the signal handler. This behavior is unexpected and seems to contradict the standard exit code practices for SIGTERM handling.

Environment

• nginx-prometheus-exporter Version: 0.8.0(The issue is reproducible in the latest version 0.11.0)
• Operating System: darwin/arm64
• Go Version: go1.20.3
• Deployment Method: Docker, ECS

Steps to Reproduce

1. Start the nginx-prometheus-exporter with the command:

   ./nginx-prometheus-exporter -nginx.scrape-uri=http://localhost:8080/server-status
   

2. Send a SIGTERM signal to the process.
3. Observe the logs and the exit code.

Expected Behavior

The exporter should exit with a code of 0 when a SIGTERM signal is received, as per the standard Unix signal handling.

Actual Behavior

The exporter logs indicate the SIGTERM signal is received and handled correctly, with the log messages showing the steps of closing the server and exiting the program with os.Exit(0). However, the actual exit code of the process is 1.

Debugging Steps

To investigate this issue, I added additional debug logging in the signal handling portion of the code. The modified signal handler code logs the process of receiving the signal, closing the server, and calling os.Exit(0):

signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, os.Interrupt, syscall.SIGTERM)
go func() {
    receivedSignal := <-signalChan
    log.Printf("Signal received: %v. Exiting...", receivedSignal)

    log.Println("Closing the server...")
    err := srv.Close()
    if err != nil {
        log.Fatalf("Error occurred while closing the server: %v", err)
    }
    log.Println("Server successfully closed")

    log.Println("Exiting the program with os.Exit(0)")
    os.Exit(0)
}()

Logs

2023/11/13 11:30:09 Signal received: terminated. Exiting...
2023/11/13 11:30:09 Closing the server...
2023/11/13 11:30:09 http: Server closed
2023/11/13 11:30:09 Server successfully closed
2023/11/13 11:30:09 Exiting the program with os.Exit(0)

Exit Code: 1 (Checked using echo $?)

Additional Context

This issue is critical as it affects the expected behavior in containerized environments, especially in orchestration platforms like Kubernetes or AWS ECS, where the exit code is used to determine the container's state.

The debug logging confirms that the signal handling path is executed as expected, but the exit code does not reflect the os.Exit(0) call. The issue persists across different environments and seems to be specific to the way SIGTERM is handled in the exporter.

Any guidance or insights into this issue would be greatly appreciated.

[knziiy]

The issue was not reproducible with the container image tagged "edge" and the binary from the latest master branch code.

% docker run --name nginx-exporter -it --rm -p 9113:9113 nginx/nginx-prometheus-exporter:edge --nginx.scrape-uri=http://1.2.3.4:8001/server-status; echo $?

This returns 0 when a TERM signal is sent.

[lucacome]

Hi @knziiy

Thanks for reporting this, it seems like this will be fixed once we release v1.0.0 (the next release), is that right?

[knziiy]

@lucacome
Thank you for your reply. I'm not sure if it's version 1.0.0, but at least as of yesterday, the issue did not reproduce in the 'edge' Docker container.

$ docker run --name nginx-exporter -it --rm -p 9113:9113 nginx/nginx-prometheus-exporter:edge --version
nginx_exporter, version edge (branch: main, revision: 0edad8ec633b84e6569d46f09ceec0d754634bd9)
  build user:       goreleaser
  build date:       2023-10-31T17:19:51Z
  go version:       go1.21.3
  platform:         linux/amd64
  tags:             unknown

If this is going to be fixed in the next release, then I would like to close this issue.

[lucacome]

Yeah what's in edge will go in 1.0.0 soon. We can keep this open until it's released (it should be soon) just to be sure.

[lucacome]

@knziiy https://github.com/nginxinc/nginx-prometheus-exporter/releases/tag/v1.0.0 is out if you want to give it a try

[knziiy]

@lucacome

I'm pleased to confirm that the issue I was experiencing has been resolved with the new 1.0.0 version of nginx-prometheus-exporter. I appreciate the efforts and the update provided.
I'll go ahead and close this issue.