List<PartitionInfo> partitions = cluster.partitionsForTopic(topic); return Math.abs(key.hashCode()) % partitions.size(); 1. 2. 前面提到的 Kafka 默认分区策略实际上同时实现了两种策略:如果指定了 Key,那么默认实现按消息键保序策略;如果没有指定 Key,则使用轮询策略。 其他分区策略 其实还有一种比较常见...
for (Map.Entry<String, List<String>> topicEntry : consumersPerTopic.entrySet()) { String topic = topicEntry.getKey(); List<String> consumersForTopic = topicEntry.getValue(); Integer numPartitionsForTopic = partitionsPerTopic.get(topic); if (numPartitionsForTopic == null) continue; for (To...
api_version=(0,10),group_id='my_group') # topic所有的partition consumer.partitions_for_topic(topic) # 构造topicPartition对象 tps = [TopicPartition(topic, p) for p in consumer.partitions_for_topic(topic)] #为consumer分配分区 consumer.assign(tps) # kafka每个分区的最新offset consumer.end_offse...
metrics(raw=False):性能指标 partitions_for(topic):返回topic所有的分区 send(topic, value=None, key=None, partition=None, timestamp_ms=None):发送数据 1.4 KafkaProductor的工具类 # -*- coding: utf-8 -*-""" Create by Mr.Hao on 2019/10/18. """importsysimportjsonimporttracebackfrommessageim...
from kafka.structs import TopicPartition consumer = KafkaConsumer('test', bootstrap_servers=['192.168.145.128:9092']) print consumer.partitions_for_topic("test") #获取test主题的分区信息 print consumer.topics() #获取主题列表 print consumer.subscription() #获取当前消费者订阅的主题 ...
kafka-python是一个python的Kafka客户端,可以用来向kafka的topic发送消息、消费消息。5.1消息生产者代码如下:第一种写法: from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers=['127.0.0.1:9092'], value_serializer=lambda m: json.dumps(m, ensure_ascii=False).encode(), ) for _id...
consumer = KafkaConsumer(group_id='123456', bootstrap_servers=['10.43.35.25:4531']) consumer.assign([TopicPartition(topic='test_rhj', partition=0), TopicPartition(topic='test_rhj', partition=1)]) print consumer.partitions_for_topic("test_rhj") # 获取test主题的分区信息 ...
client_id (str) – 客户端名称,默认值: ‘kafka-python-{version}’ request_timeout_ms (int) – 客户端请求超时时间,单位毫秒。默认值: 30000. 参考API: https://kafka-python.readthedocs.io/en/master/apidoc/KafkaClient.html brokers() 获取所有broker元数据 available_partitions_for_topic(topic) 返回...
:param KafkaServer: kafka服务器IP :param KafkaPort: kafka工作端口 :param ClientId: 生产者名称 :param Topic: 主题 """self._bootstrap_server='{host}:{port}'.format(host=KafkaServer,port=KafkaPort)self._topic=Topic self._clientId=ClientId""" ...
docker exec -it kafka bash cd /opt/kafka/bin 5. 创建Topic,分区为2,Topic name为'kafka_demo' kafka-topics.sh --create --zookeeper zookeeper:2181 \ --replication-factor 1 --partitions 2 --topic kafka_demo 6. 查看当前所有topic kafka-topics.sh --zookeeper zookeeper:2181 --list 7. 安装kaf...