Is JsonLayout allow to manipulate logevent such as PatternLayout?

[Sangaibisi]

Hello, I just wonder is there a any possible way to manipulating log event such as masking with JsonLayout. I found LogEventPatternConverter class has a format method that signed with abstract but i couldn't similiar with JsonLayout. So my question that is there any possible way to do this?

[ppkarwasz]

@Sangaibisi,

JsonLayout and other non-configurable JSON layouts are considered obsolete and they were replaced by JsonTemplateLayout.

In the JSON template layout you can reuse all pattern converters using the pattern resolver. E.g.:

{
    "$resolver": "pattern"
    "pattern": "%replace{%m}{(\\d{4})-\\d{4}-\\d{4}-(\\d{4})}{\\1-XXXX-XXXX-\\2}"
}

could mask a common credit card number format in the message.

Edit: As remarked by Gary an Volkan, this is an example. In real production code private data shouldn't even be logged.

[garydgregory]

Note that payment card numbers can be as short as 8 digits and up to 19 digits. The 16 digit format is obviously quite common.

[vy]

@ppkarwasz, I find it really dangerous to tell people to "use X to mask credit card numbers" without understanding what they mean by "masking" – as a matter of fact, @garydgregory already pointed a hole in it. I can imagine you simply wanted to demonstrate how they can leverage %replace directive of PatternLayout in combination with JsonTemplateLayout. Nevertheless, the shared example is dangerous and we don't really know what the user is asking for.

[vy]

@Sangaibisi, as @ppkarwasz indicated, please use JsonTemplateLayout instead. It provides several knobs you can tap into the rendering process; custom event resolvers, intercepting event resolver compiler, etc. Could you elaborate on "masking" you are referring to? Could you give an example of such a masking operation? Do you want to mask a particular field? Do you want to mask all occurrences of a particular value in all fields? etc. That would help us to understand and answer your question better.

[Sangaibisi]

Hi Volkan, thanks for quick reply. I am trying to log JSON objects which more likely request/responses on a Spring boot Java app using Log4J2. As you know OncePerRequestFilter class triggered by every servlet operations. For the tracing debugging information, I logged every request and response. But these endpoints might contains sensitive data like credit card or personal information which is don't allowed to store without permission.

So, I would like to mask values like "creditCardNumber" or "name" field in every object. I able to do that with PatternLayout. I wrote custom plugging class which extends LogEventPatternConverter and simple log4j2.xml configuration.

   <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%mask{%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ} [%t] %-5level %logger{36} traceId - %X{traceId} userId- %X{userId} - %msg%n}"/>
        </Console>
    </Appenders>

@Plugin(
    name = "MaskingConverter",
    category = "Converter"
)
@ConverterKeys({"mask"})
public class MaskingConverter extends LogEventPatternConverter {
    private final PatternLayout patternLayout;

    protected MaskingConverter(String[] options) {
        super("mask", "mask");
        this.patternLayout = this.createPatternLayout(options);
    }

    public static MaskingConverter newInstance(String[] options) {
        return new MaskingConverter(options);
    }

    private PatternLayout createPatternLayout(String[] options) {
        System.out.println("Options: " + Arrays.toString(options));
        if (options != null && options.length != 0) {
            return PatternLayout.newBuilder().withPattern(options[0]).build();
        } else {
            throw new IllegalArgumentException("Options for MaskingConverter are missing.");
        }
    }

    public void format(LogEvent event, StringBuilder toAppendTo) {
        String formattedMessage = this.patternLayout.toSerializable(event);
        String maskedMessage = this.maskSensitiveValues(formattedMessage);
        toAppendTo.setLength(0);
        toAppendTo.append(maskedMessage);
    }

    private String maskSensitiveValues(String message) {
        message = message.replaceAll("(?<=key=')[^']+?(?=')|(?<=\"key\":\")[^\"]+?(?=\")", "****");
        return message;
    }
}

When I was exploring log4j packages, i just figure out the "format" method doing all the things. So I thought that if is there a method in JsonLayout which is could overridable. Now, I am trying to use JsonTemplate but faced some runtime exception. I'll give feedback when able to do.

[vy]

@Sangaibisi, I think you might need to re-think your approach here. Let me elaborate on that.

Assume you have no restrictions on the schema. You might end up having documents as follows:

{"message": "foo", "request": "bar", "response": "buzz"}
{"message": {"message": "foo", "request": "bar", "response": "buzz"}}
{"message": {"message": "foo", "request": "bar", "response": "buzz", "custom-field-added-by-programmer": "doh"}}

Sounds far fetched? Certainly not. Consider the following statements:

log.info("foo");
log.info(new StringMapMessage().with("message", "foo"));
log.info(new StringMapMessage().with("message", "foo").with("custom-field-added-by-programmer", "doh"));

I can share more edge cases, but I think you get my point.

Many popular log storage solutions (e.g., Elasticsearch) would simply come to a halt with such an approach. You wouldn't want to accept every JSON structure logged by the programmer. This is a recipe for operational disaster – if not today, certainly tomorrow. I briefly covered this with an example in the message resolver documentation. Hence...

log JSON objects which more likely request/responses on a Spring boot Java app

don't do this. If you really would like to "embed" the request/response along with the log message

1. Mask whatever you want
2. Encode them (e.g., using base64)
3. Store them in a fixed field that is defined in your schema

You can achieve this by creating your own JSON Template Layout event resolvers. There you can fetch the request/response passed along with the request context and render them as you see fit.

... custom plugging class which extends LogEventPatternConverter ...

private String maskSensitiveValues(String message) {
    message = message.replaceAll("(?<=key=')[^']+?(?=')|(?<=\"key\":\")[^\"]+?(?=\")", "****");
    return message;
}

As you also probably are aware of, maskSensitiveValues() you shared is certainly not bullet-proof. 😉

Let me know if you have further questions.

[Sangaibisi]

When I am trying to use JsonTemplateLayout, I am getting this error while application starting deploy. My log4j2.xml configuration below. What could be the problem?

main ERROR Console contains an invalid element or attribute "JsonTemplateLayout"

 <?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="CustomLayout" target="SYSTEM_OUT">
            <JsonTemplateLayout eventTemplateUri="classpath:log4j2Layout.json"/>
        </Console>
    </Appenders>

    <Loggers>
        <Logger name="com.filemarket" level="INFO" additivity="false">
            <AppenderRef ref="CustomLayout"/>
        </Logger>
        <Root level="INFO">
            <AppenderRef ref="CustomLayout"/>
        </Root>
    </Loggers>
</Configuration>

[ppkarwasz]

JsonTemplateLayout is available as a separate library with Maven coordinates:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-layout-template-json</artifactId>
</dependency>