Skywalking Kafka fetcher error after upgrade 9.2.0

[skundu1234]

Skywalking OAP Kafka fetcher config:
kafka-fetcher:
selector: ${SW_KAFKA_FETCHER:default}
default:
bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:}
namespace: ${SW_NAMESPACE:""}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:30}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:3}
enableNativeProtoLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG:true}
enableNativeJsonLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG:true}
isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:false}
consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:""}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}
kafkaConsumerConfig:
enable.auto.commit: true
security.protocol: SASL_SSL
ssl.truststore.location: ${SW_KAFKA_CONSUMER_TRUSTSTORE_LOCATION:""}
ssl.truststore.password: ${SW_KAFKA_CONSUMER_CONFIG_TRUSTSTORE_PASSWORD:""}
ssl.endpoint.identification.algorithm: ""
sasl.mechanism: SCRAM-SHA-512
sasl.jaas.config: ${SW_KAFKA_CONSUMER_JAAS_CONFIG:""}

Error log:
 org.apache.skywalking.oap.server.analyzer.agent.kafka.provider.handler.MeterServiceHandler 87 [KafkaConsumer-1-thread-3] ERROR [] - [9.3.0-SNAPSHOT-dd6b7a1] handle record failed
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:641) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:890) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData.(MeterData.java:63) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData.(MeterData.java:13) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData$1.parsePartialFrom(MeterData.java:1318) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData$1.parsePartialFrom(MeterData.java:1312) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.(MeterDataCollection.java:59) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.(MeterDataCollection.java:9) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection$1.parsePartialFrom(MeterDataCollection.java:751) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection$1.parsePartialFrom(MeterDataCollection.java:745) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.parseFrom(MeterDataCollection.java:228) ~[apm-network-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at at org.apache.skywalking.oap.server.analyzer.agent.kafka.KafkaFetcherHandlerRegister.lambda$runTask$1(KafkaFetcherHandlerRegister.java:122) ~[kafka-fetcher-plugin-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]

I see that PR is also raised.
05afa98

[wu-sheng]

You are using a snapshot rather than an official release.(9.3.0-SNAPSHOT). We don't know what is your version.

And we don't know what is your old version, the Kafka messages should be cleared if you are bumping up from a very old version.

[skundu1234]

we have copied from official website ,not from docker image.We have built image and push that in our artifactory and using it.Same thing we have done in 8.9.1 and did not face any issue.

[wu-sheng]

What you did on your private env, is totally up to you. Not our responsibility.

All I can do is, I tried to download the binary from our website, you can see the jar files are using NO SNAPSHOT suffix.

What you have changed after the download, you are on your own.

[skundu1234]

Thanks,after using 9.2.0 plugin as well, we are seeing same error.

OAP error:
2023-05-11 12:50:48,003 org.apache.skywalking.oap.server.analyzer.agent.kafka.provider.handler.MeterServiceHandler 87 [KafkaConsumer-1-thread-4] ERROR [] - [9.2.0-cb8845a] handle record failed
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:129) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:641) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:890) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData.(MeterData.java:63) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData.(MeterData.java:13) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData$1.parsePartialFrom(MeterData.java:1318) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterData$1.parsePartialFrom(MeterData.java:1312) ~[apm-network-9.2.0.jar:9.2.0]
at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:889) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.(MeterDataCollection.java:59) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.(MeterDataCollection.java:9) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection$1.parsePartialFrom(MeterDataCollection.java:751) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection$1.parsePartialFrom(MeterDataCollection.java:745) ~[apm-network-9.2.0.jar:9.2.0]
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) ~[protobuf-java-3.19.4.jar:?]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.19.4.jar:?]
at org.apache.skywalking.apm.network.language.agent.v3.MeterDataCollection.parseFrom(MeterDataCollection.java:228) ~[apm-network-9.2.0.jar:9.2.0]
at org.apache.skywalking.oap.server.analyzer.agent.kafka.provider.handler.MeterServiceHandler.handle(MeterServiceHandler.java:75) ~[kafka-fetcher-plugin-9.2.0.jar:9.2.0]
at org.apache.skywalking.oap.server.analyzer.agent.kafka.KafkaFetcherHandlerRegister.lambda$runTask$1(KafkaFetcherHandlerRegister.java:122) ~[kafka-fetcher-plugin-9.2.0.jar:9.2.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]

$ cd oap-libs
$ ls
agent-analyzer-9.2.0.jar kotlinx-coroutines-core-jvm-1.5.0-native-mt.jar
animal-sniffer-annotations-1.19.jar kotlinx-coroutines-jdk8-1.5.0-native-mt.jar
annotations-13.0.jar kotlinx-coroutines-reactive-1.5.0-native-mt.jar
antlr4-runtime-4.9.2.jar LatencyUtils-2.0.3.jar
aopalliance-1.0.jar library-client-9.2.0.jar
apm-network-9.2.0.jar library-datacarrier-queue-9.2.0.jar
apollo-client-1.8.0.jar library-elasticsearch-client-9.2.0.jar
apollo-core-1.8.0.jar library-kubernetes-support-9.2.0.jar
armeria-1.16.0.jar library-module-9.2.0.jar
armeria-graphql-1.16.0.jar library-server-9.2.0.jar
armeria-graphql-protocol-1.16.0.jar library-util-9.2.0.jar
armeria-protobuf-1.16.0.jar listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
audience-annotations-0.5.0.jar log4j-api-2.17.1.jar
banyandb-java-client-0.1.0.jar log4j-core-2.17.1.jar
bcpkix-jdk18on-1.71.jar log4j-over-slf4j-1.7.30.jar
bcprov-jdk18on-1.71.jar log4j-slf4j-impl-2.17.1.jar
bcutil-jdk18on-1.71.jar log-analyzer-9.2.0.jar
brotli4j-1.7.1.jar logging-interceptor-4.9.2.jar
checker-qual-3.12.0.jar lz4-java-1.6.0.jar
classmate-1.5.1.jar meter-analyzer-9.2.0.jar
client-java-16.0.0.jar micrometer-core-1.8.5.jar
client-java-api-16.0.0.jar mvel2-2.4.8.Final.jar
client-java-proto-16.0.0.jar nacos-api-1.4.2.jar
cluster-consul-plugin-9.2.0.jar nacos-client-1.4.2.jar
cluster-etcd-plugin-9.2.0.jar nacos-common-1.4.2.jar
cluster-kubernetes-plugin-9.2.0.jar netty-buffer-4.1.77.Final.jar
cluster-nacos-plugin-9.2.0.jar netty-codec-4.1.77.Final.jar
cluster-standalone-plugin-9.2.0.jar netty-codec-dns-4.1.77.Final.jar
cluster-zookeeper-plugin-9.2.0.jar netty-codec-haproxy-4.1.77.Final.jar
kafka-fetcher-plugin-9.2.0.jar

[skundu1234]

@wu-sheng it looks like that code base change causing this regarding this PR 05afa98

[wu-sheng]

What's made is already done. There is nothing I can do.