Posts

Showing posts with the label kafka-consumer-api

Kafka not getting rid of data when setting retention.ms

Image
Clash Royale CLAN TAG #URR8PPP Kafka not getting rid of data when setting retention.ms So when I look for a way to count the messages in a topic, this one is good kafka-run-class kafka.tools.GetOffsetShell --broker-list broker1:9092,broker2:9092,broker3:9092 --topic rev-dly-upd --time -1 kafka-run-class kafka.tools.GetOffsetShell --broker-list broker1:9092,broker2:9092,broker3:9092 --topic rev-dly-upd --time -1 The only thing is, when I change the retention.ms config to retention.ms=1000 , and even check that the topic has been configured by running kafka-topics --describe --zookeeper zookeeper1:2181 --topic rev-dly-upd . I can see clearly that that config is set at 1000... retention.ms retention.ms=1000 kafka-topics --describe --zookeeper zookeeper1:2181 --topic rev-dly-upd Topic:rev-dly-upd PartitionCount:8 ReplicationFactor:3 Configs:retention.ms=1000 Topic: rev-dly-upd Partition: 0 Leader: 159 Replicas: 159,96,160 Isr: 159,96,160 Topic: rev-dly-upd Partitio...

LEADER_NOT_AVAILABLE - docker container

Image
Clash Royale CLAN TAG #URR8PPP LEADER_NOT_AVAILABLE - docker container setting up docker containers (kafka, zookeeper) and trying to (publish and) consume from another, receive the following error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"} There is no additional error indication when publishing or consuming. When testing connectivity with telnet 172.18.0.3 9092 I receive the expected result. telnet 172.18.0.3 9092 In [2]: c.list_topics() Out[2]: ClusterMetadata(9ToJF8nPQC-rCTXGxuUalw) In [3]: l = c.list_topics() In [4]: l.brokers Out[4]: {1010: BrokerMetadata(1010, 172.18.0.3:9092)} In [7]: l.orig_broker_name Out[7]: u'172.18.0.3:9092/1010' In [8]: l.topics Out[8]: {'__consumer_offsets': TopicMetadata(__consumer_offsets, 50 partitions), 'item': TopicMetadata(item, 1 partitions), 'mytopic': TopicMetadata(mytopic, 1 partitions)} In [9]: i =l.topics['item'] In [10]: i.partitions Out[10]: {0: Partiti...