Kafka命令

刘超 21天前 ⋅ 4147 阅读   编辑

1、查看broker有哪些

  方式一

zhao@zhao:~/liujichao$ echo dump | nc server-1 2181 | grep brokers

/brokers/ids/1

  方式二

zhao@zhao:~/liujichao$ /home/zhao/software/kafka_2.11-0.10.2.1/bin/zookeeper-shell.sh server-1:2181 ls /brokers/ids

Connecting to server-1:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[1]

2、查看有哪些topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper host3:2181 --list

3、创建topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh -zookeeper 10.20.252.226:2181,10.20.252.228:2181,10.20.252.229:2181 -create -topic pass -replication-factor 2  -partitions 6

-replication-factor与partitions参数解释

4、查看Kafka中的某个Topic的信息描述

zhao@zhao:~/liujichao$ /home/zhao/software/kafka_2.11-0.10.2.1/bin/kafka-topics.sh --describe --zookeeper server-1:2181 --topic topic_adx_up_request

Topic:topic_adx_up_requestPartitionCount:6ReplicationFactor:1Configs:

Topic: topic_adx_up_requestPartition: 0Leader: 1Replicas: 1Isr: 1

Topic: topic_adx_up_requestPartition: 1Leader: 1Replicas: 1Isr: 1

5、向topic产生数据

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-producer.sh --broker-list seabox1:9092 --topic TestTopic

6、offset

-- Get the consumer offsets for a topic
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group

7、Group

-- List the consumer groups known to Kafka
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list (old api)

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list (new api)

-- View the details of a consumer group
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group

8、查看topic内容

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic trigger_web_log

/web/kafka/bin/kafka-console-consumer.sh -bootstrap-server 10.7.6.25:9092 --from-beginning --topic topic_adx_bid_win

-- Get number of messages in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

-- Get the earliest offset still in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

-- Get the latest offset still in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

-- Read from __consumer_offsets
Add the following property to config/consumer.properties: exclude.internal.topics=false

bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

-- Getting the last five message of a topic
kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e

9、Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

10、删除topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper hadoop242:2181,hadoop243:2181,hadoop244:2181  --topic designer_pcAction --delete

11、两Kafka集群topic同步

    kafka-mirror-maker.sh,具体使用在V1.0中,这个功能是有问题,需要升级Kafka

12、查看指定消费组信息下面使用的topic offset 情况

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.16.0.69:9092 --group group --describe

13、重置offset

    zookeeper中,设置set /consumers/console-consumer-29012/offsets/kafka_direct/1 0


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: