Does Coraza also support network (HTTP) Logging for auditlogs?

[amsnek]

With modsecurity3 / libmodsecurity it was possible to directly send audit logs to a http endpoint (in json format). This was a massive benefit to help eleviate the amount of logfiles (when in concurrent mode) or parsing/tracking of lines in logfiles (when in serial mode).

In my case I would directly send to a http listener from filebeat (on remote or localhost)

would the following also be possible in coraza?

SecAuditLogFormat JSON
SecAuditLogType HTTPS
SecAuditLog http://<ip>:<port>

[jcchavezs]

This haven't been implemented on purpose as there are different things that have to be discussed: retry policy, batching vs not batching, timeout, etc. I am in favour to introduce this but we need to consider all that values.

…

On Tue, 13 Jun 2023, 16:48 amsnek, ***@***.***> wrote: With modsecurity it was possible to directly send audit logs to a http endpoint (in json format). This was a massive benefit to help eleviate the amount of logfiles (when in concurrent mode) or parsing/tracking of lines in logfiles (when in serial mode). would the following also be possible in coraza? SecAuditLogFormat JSON SecAuditLogType HTTPS SecAuditLog http://<ip>:<port> — Reply to this email directly, view it on GitHub <#813>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYASKHIFBT7K6NEADETLXLB4TPANCNFSM6AAAAAAZFABNFY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[amsnek]

Thanks for your quick response!

I am actually not sure how libmodsecurity currently does it but from observation I think it does a very "simple" approach as in first in/out, no retries, no batching. This means that logs/transactions will be lost if the endpoint is not available but since there can be a huge amount of transactions/traffic on busy sites (and especially the possibility for large amounts of data if things like bodies/responses etc are captured too), I would think that buffering/retries may not be a practical approach in the case of direct network logging.

Personally, for my use cases, the current behaviour suffices I need to ensure that my log endpoint is working reliably or accept that (modsecurity) logs will be lost for the duration of the outage. I will have logs from the stack above or below, just the detailed audit transaction would is gone.

Still, File based logging has a lot disadvantages too. While concurrent concurrent mode can handle a large number of transactions and can operate asynchrous, things like filebeat could not handle it. Serial Logging has limits too, especially how the file is rolled over and the amount of transactions. Additionally, not having the files on FS at all, has some advantages too, even if the drawback may be that some transactions might be lost.

Any ongoing discussion on this subject I would gladly follow!

[jcchavezs]

I don't have enough experience operating audit logs so I would rely on getting feedback from knowledgeable people like @airween and @dune73. While my decent background in observability tells me missing some logs is fine, my short background in security tells me missing audit logs isn't too good. In any case even with all measures possible you can still lose logs so that isn't my biggest concern. My biggest concern is doing one http request to a log ingestor per http request received, that isn't too good and reminds me to my days working with Zipkin PHP. In general you want to avoid to this 1-1 pattern as basically you have to guarantee same level of availability both in the service and the logger ingestion itself which could be critical if you think of ingress gateways, also it would add some overhead to the request unless we attempt to process async so I am inclined to have some sort of batching but that requires the receiver endpoint to guarantee it can receive an array of objects instead of single objects e.g. in zipkin we always received array of objects even when the object was one. Not even mentioning the overhead of doing one HTTP request every time. Would also love to get @anuraaga's feedback on this.

…

On Tue, 13 Jun 2023, 17:39 amsnek, ***@***.***> wrote: Thanks for your quick response! I am actually not sure how libmodsecurity currently does it but from observation I think it does a very "simple" approach as in first in/out, no retries, no batching. This means that logs/transactions will be lost if the endpoint is not available but since there can be a huge amount of transactions/traffic on busy sites (and especially the possibility for large amounts of data if things like bodies/responses etc are captured too), I would think that buffering/retries may not be a practical approach in the case of direct network logging. Personally, for my use cases, the current behaviour suffices I need to ensure that my log endpoint is working reliably or accept that (modsecurity) logs will be lost for the duration of the outage. I will have logs from the stack above or below, just the detailed audit transaction would is gone. Still, File based logging has a lot disadvantages too. While concurrent concurrent mode can handle a large number of transactions and can operate asynchrous, things like filebeat could not handle it. Serial Logging has limits too, especially how the file is rolled over and the amount of transactions. Additionally, not having the files on FS at all, has some advantages too, even if the drawback may be that some transactions might be lost. Any ongoing discussion on this subject I would gladly follow! — Reply to this email directly, view it on GitHub <#813 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAQ7FT24HAUTKFG2AQ3XLCCTBANCNFSM6AAAAAAZFABNFY> . You are receiving this because you commented.Message ID: ***@***.***>

[amsnek]

I fully agree that missing logs is bad, but preferable to losing stability / peak on ressources load due to buffering attempts.

I certainly would not mind using different endpoints -> for example writing directly to redis. This could use HTTP2 (streams) or even H3/QUIC in the future and thus multiplex many transactions into one tcp connection (or udp in case of quic maybe someday)

If an endpoint isnt availble, writing for filesystem as fallback would be acceptable as well.

[airween]

I don't have enough experience operating audit logs

Unfortunately I do not have it too. I use audit.log only if I'm not able to reproduce the request from the error.log to inspect it.

[tojewel]

I've been following this discussion and I wanted to share some thoughts on how OpenTelemetry could be integrated into Coraza to address some of the points raised.

The features discussed here, such as directly sending audit logs to an HTTP endpoint in JSON format, and the considerations around retry policy, batching vs not batching, and timeout, are all areas where OpenTelemetry can provide value.

OpenTelemetry is a set of APIs, libraries, agents, and instrumentation that standardizes how you collect and transfer telemetry data (metrics, logging, and tracing). It provides a unified way to collect all forms of observability data, including logs. While OpenTelemetry itself doesn't directly send logs to an HTTP endpoint, it exports logs to a collector, and the collector is responsible for sending the logs to the desired endpoint. It does support HTTP and gRPC collectors.

OpenTelemetry also provides features to handle considerations like retry policy, batching vs not batching, and timeout. For example, the OpenTelemetry collector supports batching and retrying data export.

That being said, we were planning to implement OpenTelemetry in the Coraza project. However, since we are interested in the TinyGo version of the Coraza project (coraza-proxy-wasm), there might be some additional work required to make everything that OpenTelemetry does compatible with TinyGo.

Thanks @jcchavezs for looping me in.

[jcchavezs]

While I think landing OpenTelemetry exporter would be good, I think we could also leverage http for simple use cases. @amsnek could you tell us what would be the destination for your logs and confirm if such endpoints can receive an array of objects or single objects?

[amsnek]

I am currently only using the Filebeat (http_endpoint) feature to write to ELK (ElastichSearch). This is mostly due lack of alternatives from modsecurity though -> the filebased logfiles could not be handled by filebeat (concurrent mode because too many files, one per transaction) and serial format has other issues among with how filebeat tracks file changes (copytruncate cant be used to rotate logfiles since filebeat will observe the inodes) and complicated workarounds need to be used.

Initially Modsecurity had once planned to also be able to support redis as backend: owasp-modsecurity/ModSecurity#1139

-> this never came to pass, but i t hink it would have been one of the best solutions for a wide array of use cases.

But to answer the Question. Currently using: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-http_endpoint.html

"The HTTP Endpoint input initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either an object or an array of objects."

[jcchavezs]

Thanks for the info. I think for retrocompatibility sake we should provide one HTTP exporter.

[amsnek]

what would be very important, if possible, that corraza supports rotating the auditfile by creating a new logfile. this is currently not possible with modsecurity and causes isses with file shippers that track the inode of the logfile (e.g filebeat)
see also owasp-modsecurity/ModSecurity-nginx#121
If we could prevent compatibility with this issue, that would be most awesome.

[jptosso]

I have personally tested audit logs with logrotate and it works fine, maybe we should add docs to coraza.io

[amsnek]

as long as it works without copytruncate from logrotate, it is fine.

[jcchavezs]

Any movement on this? Shall we formalize the https inclusion for retrocompatibility as an issue?

[amsnek]

For retrocompatibility I would think it would be a good idea. I doubt I am the only one using this though I dont have any data on this.
I did not realise that I shouldt take further actions at this point in time -> shall I create an issue on this?

[Nihilistttt]

If you are interested diffrent auditlog source, I found forked repo
https://github.com/Eyup-Devop/coraza
using kafka for producing auditlog

[jcchavezs]

Thanks for bringing this to our attention. I strongly encourage you to work with us to support such kind of exporters through plugins rather than using a fork. Are you up to that?

…

On Wed, Sep 6, 2023 at 8:05 PM Nihilistttt ***@***.***> wrote: If you are interested diffrent auditlog source, I found forked repo https://github.com/Eyup-Devop/coraza using kafka for producing auditlog — Reply to this email directly, view it on GitHub <#813 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAQMSJ4MXAXSDQFZBM3XZC3QPANCNFSM6AAAAAAZFABNFY> . You are receiving this because you were mentioned.Message ID: ***@***.***>