Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continuous restart of Kafka Produce Consume Test pod #292

Open
shubham-pandey95 opened this issue Sep 27, 2019 · 0 comments
Open

Continuous restart of Kafka Produce Consume Test pod #292

shubham-pandey95 opened this issue Sep 27, 2019 · 0 comments

Comments

@shubham-pandey95
Copy link

shubham-pandey95 commented Sep 27, 2019

First of all, thank you very much for putting this repo together. I used this repo to setup Kafka and the only thing I changed was to introduce podAntiAffinity and provisioning GCE disks and PVC beforehand rather than dynamic volume provisioning. Zookeeper and Kafka pod installation is working fine but when trying to run the tests, the produce-consume pod is facing continuous restarts:

NAME                       READY   STATUS    RESTARTS   AGE
kafkacat-sz4vz             3/3     Running   0          20h
produce-consume-zcwlz      3/3     Running   336        20h
replication-config-wckl5   1/1     Running   0          20h

After a little debugging, I found the issue to be coming from the producer container and these are logs for the pod before getting killed:

kubectl logs -f produce-consume-zcwlz -c producer -n test-kafka --previous
++ date --iso-8601=ns -u
+ echo '--- start produce-consume-zcwlz 2019-09-27T18:01:37,693687068+00:00 ---'
+ ./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --describe --topic test-produce-consume
Topic:test-produce-consume	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: test-produce-consume	Partition: 0	Leader: 0	Replicas: 0,2	Isr: 0,2
	Topic: test-produce-consume	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1
	Topic: test-produce-consume	Partition: 2	Leader: 1	Replicas: 1,0	Isr: 1,0
+ tail -f /shared/produce.tmp
+ ./bin/kafka-console-producer.sh --broker-list bootstrap.kafka:9092 --topic test-produce-consume --producer-property acks=-1
/bin/bash: line 1:   324 Broken pipe             tail -f /shared/produce.tmp
       325 Killed                  | ./bin/kafka-console-producer.sh --broker-list $BOOTSTRAP --topic test-produce-consume --producer-property "acks=$ACK

I increased the resource limits for this container to 500m and 800Mi but this issue is still present. Can't find anything substantial in the zookeeper and kafka logs. Any idea what could be causing this? Or any metric that I should check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant