Usage of BaseProcessor for filtering logs

[sunildatla]

Hi ,
I am looking to filter the logs (like sampling after a certain count is reached in a minute) with opentelemetry and thinking to use BaseProcessor's OnEnd method to call base.OnEnd(data) . Is this the right place to customize for sampling the logs for example , What is the purpose of baseprocessor otherwise . I tried to filter based on eventid but noticed that the logs were still exported to exporters even though base.OnEnd(data) is not called.

Any suggestions or recommendations

[cijothomas]

Can you show the full code of the processor you have and how was it wired up? (I agree we have a doc gap to show LogFiltering)

[sunildatla]

Its a pretty straight forward Processor as it is a POC .

Also is Using Processor a right approach for filtering logs based on its usage ? I can use log filters in config but that will remove a particular level or category all at once , here we want to control the amount of logs .
`
internal class LogFilteringProcessor : BaseProcessor
{
public LogFilteringProcessor() { }

    public override void OnEnd(LogRecord data)
    {
        **if(data.EventId==1)
        base.OnEnd(data);**

    }
   
}`

Below is the startup 


serviceCollection
                       .AddLogging(builder => builder
                       .AddFilter("*", LogLevel.Trace)
                       .AddProvider(new IntuneOtelLoggerProvider(logConfigOptionsMonitor))
                       **.AddProcessor(new LogFilteringProcessor())**
                       .AddOpenTelemetryLogging(options =>
                       {
                           options.UseFormattedMessage = true;
                           options.IncludeStackTrace = true;
                           options.IncludeScopes = true;
                       })
                       .AddGenevaExporter(genevaOptions =>
                       {
                           genevaOptions.ConnectionString = "EtwSession=OpenTelemetrySession";
                           genevaOptions.TableNameMappings = new Dictionary<string, string>
                           {
                               ["LoggingBenchmarks.Loggers.LoggerCategory"] = "IntuneEventnew"
                               //["*"]="IntuneEvent"
                           };
                       })
                       .AddConsoleExporter())

[cijothomas]

^ wont work. The FilteringProcessor wont have any impact on the AddConsole and AddGeneva as they use their own processors, which are called irrespective of FilteringProcessor. The only way would be to manually construct the pipeline, and instantiate the Exporters by hand (not feasible today for GenevaExporter which has no public ctor))

As of this is an limitation for any Exporters which do not offer public ctors:
See #3918 (comment)

[sunildatla]

@cijothomas is that the case for traces also ? Because i was trying to use a compositeProcessor like below sometime back and i believe it was able to filter the traces without sending to geneva

internal class FilteringProcessor : CompositeProcessor
{
private readonly Func<Activity, bool> filter;
public FilteringProcessor(BaseProcessor processor ,Func<Activity,bool> filter) : base(new[] { processor })
{
this.filter = filter ?? throw new ArgumentNullException(nameof(filter));
}
public override void OnEnd(Activity data)
{
if (this.filter(data))
base.OnEnd(data);
}
}

Startup:

.AddProcessor(new FilteringProcessor(new CommonTagProcessor("BasicProcessor"),(act)=>true))

[sunildatla]

Also is there any other way i can filter or eliminate logs in the application without sending it to geneva or any other exporters , if processor doesnt support it.

[cijothomas]

For traces this would work for any exporter: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/extending-the-sdk/MyFilteringProcessor.cs

The one you showed above wont work for Exporters like Geneva which do not have public ctor.