Log sending delay

[YouZhengChuan]

Gostash always sends old logs. The latest logs always wait a few minutes before they start to send, and the longer gogstash runs, the longer the log delay will be, and the delay will be several days at most.

[tsaikd]

What's your input/filter/output?

[YouZhengChuan]

input:
  - type: file
    # (required)
    path: "/home/work/log/nginx/pcdn_access.log"
    # (optional), one of ["beginning", "end"], default: "end"
    start_position: end
    # (optional), default: ".sincedb.json"
    sincedb_path: ".sincedb.json"
    # (optional), in seconds, default: 15
    sincedb_write_interval: 15

filter:
  - type: grok
    match: ["%{NGINX_ACCESS}"]
    source: "message"
    patterns_path: "configs/grok-patterns.grok"

  - type: grok
    match: ["%{NGINX_ACCESS_REQUEST}"]
    source: "rawrequest"
    patterns_path: "configs/grok-patterns.grok"

  - type: grok
    match: ["%{REQUEST_URL}"]
    source: "request_url"
    patterns_path: "configs/grok-patterns.grok"

  - type: url_param
    source: "request_param"
    prefix: "param_"

  - type: grok
    match: ["%{STATUS_N}"]
    source: "status"
    patterns_path: "configs/grok-patterns.grok"
    #patterns:

  - type: date
    format: ["02/Jan/2006:15:04:05 +0800"]
    source: time_local
    joda: false

  - type: typeconv
    conv_type: int64
    fields: ["body_bytes_sent", "status", "server_port"]

  - type: typeconv
    conv_type: float64
    fields: ["httpversion", "request_time", "upstream_response_time"]

  - type: add_field
    key: "metric"
    value: "nginx_alog"

  - type: add_field
    key: "module_name"
    value: "sd01-db5-nginx"

  - type: remove_field
    fields: ["rawrequest", "request_url", "request_param", "message"]

output:
  - type: http
    urls: ["http://10.20.0.13:8090/api/v1/log"]
  #- type: stdout

[YouZhengChuan]

configs/grok-patterns.grok:

RANGE %{WORD}=[0-9]*-[0-9]*
HTTPRANGE (%{RANGE}|-)
NGINX_ACCESS %{IPORHOST:remote_addr} (?:-|(%{WORD}.%{WORD})) %{USER:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:rawrequest}" %{NUMBER:status} (?:%{NUMBER:body_bytes_sent}|-) (?:%{QS:http_refere}|-) %{QS:http_user_agent} "%{HTTPRANGE:http_range}" %{NUMBER:request_time} (%{NUMBER:upstream_response_time}|-)
NGINX_ACCESS_REQUEST (?:%{WORD:rq_method}) (%{NOTSPACE:request_url}) (%{HOSTPORT:upstream_addr}|-) (HTTP/%{NUMBER:httpversion})

# $scheme://$host:$server_port$request_uri
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\\\-\[\]<>]*
REQUEST_USER (?:%{USER}(?::[^@]*)?@)?
REQUEST_URL %{WORD:scheme}://%{REQUEST_USER:request_user}%{IPORHOST:host}:%{NUMBER:server_port}%{URIPATH:request_path}(%{URIPARAM:request_param}|)

# status  200,201 -> 2
HTTP_CODE_N [0-9]
STATUS_N %{HTTP_CODE_N:status_n}

[YouZhengChuan]

@tsaikd
it is possible that there is a packet loss caused by the network between the server where gogStash and the "Http Api".

"out put" is sent asynchronously when sending data? If a certain data transmission delay, will affect the subsequent data transmission?

[tsaikd]

Output events running in goroutines for each output module. But main output goroutine will process the next event until all events processed.
https://github.com/tsaikd/gogstash/blob/master/config/output.go#L77-L87

If you can reproduce, try to add output/report for debugging.