采用kop插件消费pulsar慢问题

Viewed 39

image.png
ck只支持kafka协议,运行一段时间就变慢, ck日志会一直打印 StorageKafka (): Can't get assignment. Will keep trying,消费就很慢,一直在尝试消费

版本:

  • pulsar:3.0.6
  • kop:3.0.6
2025-03-12T10:29:00,580+0800 [metadata-store-10-1] ERROR io.streamnative.pulsar.handlers.kop.storage.PartitionLogManager - Recovery of persistent://xxx-11 failed
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.NotLeaderOrFollowerException
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
        at io.streamnative.pulsar.handlers.kop.KafkaTopicLookupService.handleGetTopicException(KafkaTopicLookupService.java:102) ~[?:?]
        at io.streamnative.pulsar.handlers.kop.KafkaTopicLookupService.lambda$getTopic$1(KafkaTopicLookupService.java:50) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
        at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$existsFromStore$11(ZKMetadataStore.java:362) ~[org.apache.pulsar-pulsar-metadata-3.0.6.jar:3.0.6]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.111.Final.jar:4.1.111.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.kafka.common.errors.NotLeaderOrFollowerException
3 Answers

这个应该是开源KoP版本的问题,没有正确处理相关的异常并返回给Client。可以通过重启Client临时解决。

出现这个异常,怎么处理,告诉client重新连接呢

重启Client,我client是ck,重启一下,一会又出现,不能完全解决

这是一个已知的开源 KoP Bug,需要自行修复下。

close 掉这个问题了。