DLQ with start_timestamp is not working correctly

[ShivanshuBagga2]

Hi,

• Version: 6.0.0
• Operating System: CentOS 7

Following is my configuration for DLQ input plugin-

input{
 dead_letter_queue {
    path => "/opt/shivanshu/setup/logstash-6.0.0/data/dead_letter_queue"
    start_timestamp => "2026-02-08T06:10:21"
    #commit_offsets => false
    pipeline_id => "main"
    tags => "dlq"
 }
}


filter{
}

output
{

if!("dlq" in [tags]){
file{
gzip => true
path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/logstash_%{+YYYY-MM-dd}.log.gz"
}
}else{
file{
gzip => true
path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/dlq-logstash_%{+YYYY-MM-dd}.log.gz"
}
}

I have some events stored in DLQ and wanna process events only those are added recently. So I use start_timestamp , but even after apply this I'm getting all events . So for testing purpose I changed its value to 2026-02-08T06:10:21, but even in this case I'm getting all events .

Here is my one sample event-

{"Locale":"Desiredresponselocalization215950","Message":{},"EventType":"Usage215940","ResourceName":"/api/startSessionForPerson##","Time":"2018-02-08T01:10:21.595-0500","UserAction":"userActionData1518070221594","SubjectType":"YYTU","ResponsePayload":"responsePayloadData1518070221594","@version":1,"host":"127.0.0.1","UserAgent":"userAgentData1518070221594","ClientId":695,"logger_name":"DemoApplication","CorrelationId":"4861cb44-0921-4da7-21594082ec-e86dcc05be35215940","SubjectId":"59bc52da6fb36c95771797f1215950","ElapsedTime(ms)":1964106054,"SessionId":"DB8BA6C6F897136A533696E84D681AB4215950","ModelId":"modelIdData1518070221594","RequestPayload":"requestPayloadData1518070221594","Status":"MOVED_PERMANENTLY","AdminId":"08340215940","ParentSpanID":"e86dcc05be35-0921-5431cb44-82ec-4da7215950","level":"INFO","Lifecycle":"qw","count":4,"Host":"aaaaaaa.hewitt.com","ResourceVersion":5.5,"Severity":"severityData1518070221594","RoleCd":"roleCdData1518070221594","SrcIP":"10.100.19.23","tags":["dlq"],"@timestamp":"2018-02-08T06:10:21.638Z","ConsumerRefID":"ddbdb1bc-b826-4265-9163-3e81e3712e64215940","SystemInstanceId":"5431cb44-8654-4da7-82ec-e86dcc05be35215950","port":42742,"thread_name":"pool-1-thread-2","level_value":20000,"ConsumerId":"Portal_44","ResourceInstance":"42272_CL1_TBAWS_QC_M1215950","@metdata":{"ip_address":"127.0.0.1"},"externalServiceName":"externalServiceNameData1518070221594","TestCfg":"JJ#UY","RequestType":"POST","SpanID":"5431cb44-0921-4da7-82ec-e86dcc05be35215950","EffectiveDate":"02-08-2018"}

You can find same question at here and here

[jordansissel]

Please do not file the same issue in multiple places.

…

On Fri, Feb 9, 2018 at 12:29 AM ShivanshuBagga2 ***@***.***> wrote: Hi, - Version: 6.0.0 - Operating System: CentOS 7 Following is my configuration for DLQ input plugin- input{ dead_letter_queue { path => "/opt/shivanshu/setup/logstash-6.0.0/data/dead_letter_queue" start_timestamp => "2026-02-08T06:10:21" #commit_offsets => false pipeline_id => "main" tags => "dlq" } } filter{ } output { if!("dlq" in [tags]){ file{ gzip => true path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/logstash_%{+YYYY-MM-dd}.log.gz" } }else{ file{ gzip => true path => "/opt/shivanshu/setup/logstash-6.0.0/shivanshu/dlq-logstash_%{+YYYY-MM-dd}.log.gz" } } I have some events stored in DLQ and wanna process events only those are added recently. So I use start_timestamp , but even after apply this I'm getting all events . So for testing purpose I changed its value to 2026-02-08T06:10:21, but even in this case I'm getting all events . Here is my one sample event- ***@***.***":1,"host":"127.0.0.1","UserAgent":"userAgentData1518070221594","ClientId":695,"logger_name":"DemoApplication","CorrelationId":"4861cb44-0921-4da7-21594082ec-e86dcc05be35215940","SubjectId":"59bc52da6fb36c95771797f1215950","ElapsedTime(ms)":1964106054,"SessionId":"DB8BA6C6F897136A533696E84D681AB4215950","ModelId":"modelIdData1518070221594","RequestPayload":"requestPayloadData1518070221594","Status":"MOVED_PERMANENTLY","AdminId":"08340215940","ParentSpanID":"e86dcc05be35-0921-5431cb44-82ec-4da7215950","level":"INFO","Lifecycle":"qw","count":4,"Host":" aaaaaaa.hewitt.com ***@***.******@***.***":{"ip_address":"127.0.0.1"},"externalServiceName":"externalServiceNameData1518070221594","TestCfg":"JJ#UY","RequestType":"POST","SpanID":"5431cb44-0921-4da7-82ec-e86dcc05be35215950","EffectiveDate":"02-08-2018"} You can find same question at here <https://discuss.elastic.co/t/dlq-with-start-timestamp-is-not-working-correctly/118980> and here <logstash-plugins/logstash-input-dead_letter_queue#17> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9125>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6vDVaNv7h8e1KFyrfkd4Q2vtbz-Aks5tTAIEgaJpZM4R_kxy> .

[ShivanshuBagga2]

Anyone looking this issue ?

[andrewvc]

We will try and take a look into this soon, apologies for the delay.

[yaauie]

See answer in: logstash-plugins/logstash-input-dead_letter_queue#17

[original-brownbear]

@ShivanshuBagga2 @yaauie I'm going to close here. I tested this manually and read over our tests, the bug here doesn't seem to be in LS core (the filter by date works in tests here). Let's continue this in the DLQ plugin issue linked above.