Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using simply a grok output an error #79

Open
le-ross opened this issue Jan 20, 2021 · 26 comments
Open

Using simply a grok output an error #79

le-ross opened this issue Jan 20, 2021 · 26 comments

Comments

@le-ross
Copy link

le-ross commented Jan 20, 2021

Installed the app globaly and LOKI running locally

at first had this

input {
  file {
    path => "./logs/*.log"
  }
}

output {
  loki {
    host => localhost
    port => 3100
    path => "/loki/api/v1/push"
  }
}

and each time I insert logs I get [Wed, 20 Jan 2021 16:34:15 GMT] ERROR error Wrong HTTP Post return code: 400

So I taught maybe I need to parse better each line with this :

filter { 
  grok { 
    match => '%{TIMESTAMP_ISO8601:timestamp}: %{GREEDYDATA:message}' 
  }
}

Now I only get this error of loading module.

Loading npm module... @pastash/filter_grok
[Wed, 20 Jan 2021 16:16:13 GMT] ERROR Unable to load urls from command line
[Wed, 20 Jan 2021 16:16:13 GMT] ERROR TypeError: Cannot read property 'create' of undefined
    at LogstashAgent.configure (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:143:21)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:215:10)
    at C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:3096:16
    at replenish (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:998:17)
    at iterateeCallback (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:983:17)
    at C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\node_modules\async\dist\async.js:958:16
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:265:5)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:220:7)
    at LogstashAgent.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\agent.js:156:9)
    at FilterAddVersion.<anonymous> (C:\Users\LeChat\AppData\Roaming\npm\node_modules\@pastash\pastash\lib\lib\base_filter.js:36:5)

I dont have the easier time using that application. what I am missing

@lmangani
Copy link
Member

lmangani commented Jan 20, 2021

Could you paste your full pastash config? Which version of Loki are you running? (or is it the one above?)

@lmangani
Copy link
Member

The grok filter is indeed no longer part of the default build - but that's a parallel issue - please post the loki errors and/or capture the socket traffic with ngrep -W byline port 3100 -d any and attach the output during the failure.

@lmangani
Copy link
Member

I also suspect your target system might lack the build tools required to compile the grok module - this might cause the 2nd failure.

@le-ross
Copy link
Author

le-ross commented Jan 20, 2021

I started to work with promtail to see if I can make it work and now I suspect it's the same error 400

promtail_1 | level=error ts=2021-01-20T20:06:26.4438654Z caller=client.go:305 component=client host=loki:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry for stream '{filename=”/var/log/AAAAAA.log", job=“varlogs”}’ has timestamp too old: 2020-12-19 12:54:30 +0000 UTC"

I dont understand why this error. It's weird that I cannot ingest old logs.

For the second error I am still looking.

@le-ross
Copy link
Author

le-ross commented Jan 20, 2021

Am I supposed to install like the output_loki? because I dont see on npm or the in repo.

$ npm install --unsafe-perm -g @pastash/filter_grok npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found npm ERR! 404 npm ERR! 404 '@pastash/filter_grok@latest' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\LeChat\AppData\Roaming\npm-cache\_logs\2021-01-20T23_24_40_297Z-debug.log

@le-ross
Copy link
Author

le-ross commented Jan 20, 2021

This is the logs when I install pastash. I dont see any reference for filters or grok

pastash logs.txt

@lmangani
Copy link
Member

@le-ross I must be honest i never tried running this on windows. In either case, you do not need grok, the error is elsewhere.
Please run ngrep -W byline port 3100 -d any and while running. Try inserting data into Loki (either way). Attach the output.

@le-ross
Copy link
Author

le-ross commented Jan 21, 2021

I will need to install ubuntu becase I am not able to find ngrep for windows

I tried to wrap the package in a docker container like this

FROM node:12

RUN npm install -g @pastash/pastash @pastash/output_loki

WORKDIR /app

CMD pastash --config_file=./pastash_loki.json

It doesnt make sense that I keep having this filter_grok error also. are you sure it's not missing somewhere? I need that grok otherwise all my logs are insert with today's timestamp

I was able to insert the logs with graylog instead of loki to see that all I am missing is the grok filter

@le-ross
Copy link
Author

le-ross commented Jan 21, 2021

tried under my ubuntu session. same grok error.

[Thu, 21 Jan 2021 20:03:38 GMT] NOTICE Starting pastash 1.0.71
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Max http socket 100
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config file : ./pastash-config.json
[Thu, 21 Jan 2021 20:03:38 GMT] INFO File loaded, 4 urls found
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config : 7 urls
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output Gelf
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to  Gelf localhost:12201
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output file
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to file ./tmp/file.log
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddHost
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddTimestamp
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR Unable to load urls from command line
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR TypeError: Cannot read property 'create' of undefined
    at LogstashAgent.configure (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
    at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16
    at replenish (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17)
    at iterateeCallback (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17)
    at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:265:5)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:220:7)
    at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:156:9)
    at FilterAddVersion.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)

@lmangani
Copy link
Member

Do you have build-essential meta package installed?

@lmangani
Copy link
Member

Btw this still does not explain the error you're receiving you're receiving:

each time I insert logs I get [Wed, 20 Jan 2021 16:34:15 GMT] ERROR error Wrong HTTP Post return code: 400

You forgot to run the ngrep command to see what we're sending out. Is the date on your server in sync with the world?

@le-ross
Copy link
Author

le-ross commented Jan 21, 2021

Oh yeah, got distracted when I saw a image ready to use. I don't have the problem of grok with that image which is version 1.0.50

here is the log with a dummy file also attached
dummy.log

interface: any
filter: ( port 3100 ) and (ip || ip6)
####################################
T 127.0.0.1:50314 -> 127.0.0.1:3100 [AP] #36
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
########
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #44
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
#
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #45
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
###
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #48
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

#
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #49
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

###
T 127.0.0.1:3100 -> 127.0.0.1:50314 [AP] #52
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.

##########

@le-ross
Copy link
Author

le-ross commented Jan 21, 2021

here is my docker-compose in case

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - 3100:3100
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:latest
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

@lmangani
Copy link
Member

204 is good. From the logs it looks as if they were accepted

HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.

@le-ross
Copy link
Author

le-ross commented Jan 22, 2021

manged to get ngrep with the error 400

interface: any
filter: ( port 3100 ) and (ip || ip6)
#######
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #7
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket:  -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket:  -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
#
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #8
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket:  -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket:  -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
###
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #11
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER

#
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #12
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER

########^Cexit

@le-ross
Copy link
Author

le-ross commented Jan 22, 2021

it's enough to throw that error with that data

2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] A
2020-09-12T01:57:50: 12 Sep 01:57:49 - [info] B
2020-09-12T01:57:51: 12 Sep 01:57:49 - c
2020-09-12T01:57:52: 12 Sep 01:57:49 d
2020-09-12T01:57:53: 12 Sep 01:e
2020-09-12T01:57:54: 12 Sef
2020-09-12T01:57:55: g

@spady7
Copy link

spady7 commented Aug 2, 2023

I get this error too. Is this resolved?
[Wed, 02 Aug 2023 09:51:09 GMT] NOTICE Starting pastash 1.0.75 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Max http socket 100 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config file : /opt/pastash/sonus.conf [Wed, 02 Aug 2023 09:51:09 GMT] INFO File loaded, 7 urls found [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config : 10 urls [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing output HEP/EEP Server [Wed, 02 Aug 2023 09:51:09 GMT] INFO Start output to HEP udp to 127.0.0.1:9060 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddHost [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddTimestamp [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddVersion [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter Multiline [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initialized multiline filter with start_line_regex: /^\[\d{4}-\d{2}-\d{2}/, flags: Loading npm module... @pastash/filter_grok [Wed, 02 Aug 2023 09:51:09 GMT] ERROR Unable to load urls from command line [Wed, 02 Aug 2023 09:51:09 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create') at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16 at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17) at iterateeCallback (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16 at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:265:5) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:220:7) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:156:9) at FilterMultiline.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)

@Dletta
Copy link
Collaborator

Dletta commented Aug 2, 2023 via email

@spady7
Copy link

spady7 commented Aug 2, 2023

Hi @Dletta i need to use "pastash" to parse logs from Sonus/Ribbon SBC.
i took configuration from wiki. Btw this is my conf:

`input {
file {
path => "/tmp/webui.log"
}
}

filter {
multiline {
start_line_regex => /^[\d{4}-\d{2}-\d{2}/
}
grok {
match => '(?m)[%{TIMESTAMP_ISO8601:timestamp}] %{WORD:pid} %{WORD:seq} \n%{GREEDYDATA:payload}'
}
regex {
regex => /From:.@(.):(\d+)/
fields => [srcIp,srcPort]
}
regex {
regex => /To:.@(.):(\d+)/
fields => [dstIp,dstPort]
}
regex {
regex => /Call-I.: (.)/
fields => [correlation_id]
}
}

output {
if [tags] != "_grokparsefailure" {
hep {
host => '127.0.0.1'
port => 9060
hep_id => 2222
hep_type => 1
}
}
}`

And when i try

pastash --config_file=/opt/pastash/sonus.conf

i get errors.

@spady7
Copy link

spady7 commented Aug 21, 2024

Has anyone found a solution to this problem?
Get same error and If i try to install as plugin i get:

npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found
npm error 404
npm error 404  '@pastash/filter_grok@*' is not in this registry.
npm error 404
npm error 404 Note that you can also install from a
npm error 404 tarball, folder, http url, or git url.
npm error A complete log of this run can be found in: /root/.npm/_logs/2024-08-21T10_34_04_889Z-debug-0.log

@Dletta
Copy link
Collaborator

Dletta commented Aug 21, 2024

@spady7 This may occur if your path definition is insufficient.

Could you run pastash from the /usr/lib/node_modules/@pastash/pastash directory and see if it still occurs?

@spady7
Copy link

spady7 commented Aug 21, 2024

@Dletta just tried, but is the same:

root@d9c9a162c2e3:/opt# /usr/local/lib/node_modules/\@pastash/pastash/bin/pastash --config_dir=/opt
[Wed, 21 Aug 2024 14:50:13 GMT] NOTICE Starting pastash 1.0.82
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Max http socket 100
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Loading config files from : /opt
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Files loaded from directory, 4 urls found
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Loading config : 7 urls
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing output Stdout
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing output file
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddHost
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddTimestamp
[Wed, 21 Aug 2024 14:50:13 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Wed, 21 Aug 2024 14:50:13 GMT] ERROR Unable to load urls from command line
[Wed, 21 Aug 2024 14:50:13 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1014:17)
    at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:255:8)
    at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:204:10)

If i use others filters they work. For instance app_audiocodes it working.

@spady7
Copy link

spady7 commented Aug 22, 2024

Hi @Dletta
I am investigating the problem thoroughly. First of all, I am using pastash in a docker container, where I have the problem. If I install pastash directly on a host machine (debian) everything works correctly. If I use the container, I have the problem described.
I understood that when pastash starts, it loads the filters, if they are specified in the conf file, from the following folder:
/usr/local/lib/node_modules/@pastash/pastash/lib/filters where there are all the "filter_*.js" filters.
I tried, by modifying the configuration file used by pastash, to load other filters, eg "csv", "grep", etc and it always works. For example, if I rename the file "/usr/local/lib/node_modules/@pastash/pastash/lib/filters/filter_csv.js" to "/usr/local/lib/node_modules/@pastash/pastash/lib/filters/filter_csv_ORI.js" pastash gives me an error when it starts (rightly so because it can't find the specified file.
That said, if I try to load only the grok filter I continue to have the problem.
How is this possible?

@Dletta
Copy link
Collaborator

Dletta commented Aug 22, 2024

@spady7

Can you share your docker compose or Dockerfile, I 'd like to try myself

@spady7
Copy link

spady7 commented Aug 22, 2024

@Dletta Sure, see following:

Dockerfile

FROM node:lts-alpine

# Install jq
RUN apk update && apk add --no-cache jq

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/
COPY install-global-packages-alpine.sh /usr/src/app/

# Ensure the script is executable
RUN chmod +x install-global-packages-alpine.sh

# Run the script to install global npm packages
RUN ./install-global-packages-alpine.sh

RUN mkdir -p /config

EXPOSE 9999/tcp 514/udp 9999/udp 514/tcp
CMD [ "/usr/local/bin/pastash", "--config_dir", "/config" ]

Here a I used alpine node container as base. But tried also with "node:20-slim" but same issue.
Let me know.
Thanks

@spady7
Copy link

spady7 commented Aug 22, 2024

@Dletta FYI i tried making container starting from qxip's official image (https://hub.docker.com/r/qxip/pastash/tags) and i get same error.
2024-08-22 15 37 37

root@47e8f2dc2cc7:/usr/src/app# node bin/pastash --log_level=debug --config_dir /config
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Changing log_level debug
[Thu, 22 Aug 2024 13:39:57 GMT] NOTICE Starting pastash 1.0.81
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Max http socket 100
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Loading config files from : /config
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Files loaded from directory, 4 urls found
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Loading config : 7 urls
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Loading urls [
  'filter://add_host://',
  'filter://add_timestamp://',
  'filter://add_version://',
  'input://udp://?host=0.0.0.0&port=30520&tags=CDR',
  'filter://grok://?match=%25%7BTIMESTAMP_ISO8601%3Atimestamp%7D%3A%20%25%7BGREEDYDATA%3Amessage%7D',
  'output://stdout://',
  'output://file://?path=%2Fusr%2Fsrc%2Fapp%2Foutput_testing.json&serializer=json_logstash'
]
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module output
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing output Stdout
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module output
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing output file
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Start output to file /usr/src/app/output_testing.json
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddHost
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddTimestamp
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] INFO Initializing filter AddVersion
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG Initializing module filter
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG no local module found filters filter_grok Error: Cannot find module 'filters/filter_grok'
Require stack:
- /usr/src/app/lib/agent.js
- /usr/src/app/bin/pastash
    at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:133:17)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/usr/src/app/lib/agent.js', '/usr/src/app/bin/pastash' ]
}
Loading npm module... @pastash/filter_grok
[Thu, 22 Aug 2024 13:39:57 GMT] DEBUG NestedError: Could not require module '@pastash/filter_grok'
    at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:14:11)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/src/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/src/app/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/src/app/lib/agent.js:255:8)
Caused By: TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
    at validateString (node:internal/validators:162:11)
    at Module.require (node:internal/modules/cjs/loader:1228:3)
    at require (node:internal/modules/helpers:176:18)
    at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:12:12)
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
  nested: TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
      at validateString (node:internal/validators:162:11)
      at Module.require (node:internal/modules/cjs/loader:1228:3)
      at require (node:internal/modules/helpers:176:18)
      at requireg (/usr/src/app/node_modules/requireg/lib/requireg.js:12:12)
      at LogstashAgent.configure (/usr/src/app/lib/agent.js:139:18)
      at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
      at /usr/src/app/node_modules/async/dist/async.js:3113:16
      at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
      at /usr/src/app/node_modules/async/dist/async.js:1019:9
      at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24) {
    code: 'ERR_INVALID_ARG_TYPE'
  }
}
[Thu, 22 Aug 2024 13:39:57 GMT] ERROR Unable to load urls from command line
[Thu, 22 Aug 2024 13:39:57 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create')
    at LogstashAgent.configure (/usr/src/app/lib/agent.js:143:21)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:215:10)
    at /usr/src/app/node_modules/async/dist/async.js:3113:16
    at replenish (/usr/src/app/node_modules/async/dist/async.js:1014:17)
    at /usr/src/app/node_modules/async/dist/async.js:1019:9
    at eachLimit$1 (/usr/src/app/node_modules/async/dist/async.js:3199:24)
    at Object.<anonymous> (/usr/src/app/node_modules/async/dist/async.js:1049:16)
    at LogstashAgent.start_modules (/usr/src/app/lib/agent.js:214:9)
    at LogstashAgent.start_filters (/usr/src/app/lib/agent.js:255:8)
    at LogstashAgent.<anonymous> (/usr/src/app/lib/agent.js:204:10)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants