Kafka命令

刘超 11天前 ⋅ 4604 阅读   编辑

  基于kafka0.10.2测试

1、查看broker有哪些

  方式一

zhao@zhao:~/liujichao$ echo dump | nc server-1 2181 | grep brokers

/brokers/ids/1

  方式二

zhao@zhao:~/liujichao$ /home/zhao/software/kafka_2.11-0.10.2.1/bin/zookeeper-shell.sh server-1:2181 ls /brokers/ids

Connecting to server-1:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[1]

2、查看有哪些topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper host3:2181 --list

3、创建topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh -zookeeper 10.20.252.226:2181,10.20.252.228:2181,10.20.252.229:2181 -create -topic pass -replication-factor 2  -partitions 6

-replication-factor与partitions参数解释

4、查看Kafka中的某个Topic的信息描述

zhao@zhao:~/liujichao$ /home/zhao/software/kafka_2.11-0.10.2.1/bin/kafka-topics.sh --describe --zookeeper server-1:2181 --topic topic_adx_up_request

Topic:topic_adx_up_requestPartitionCount:6ReplicationFactor:1Configs:

Topic: topic_adx_up_requestPartition: 0Leader: 1Replicas: 1Isr: 1

Topic: topic_adx_up_requestPartition: 1Leader: 1Replicas: 1Isr: 1

5、向topic产生数据

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-producer.sh --broker-list seabox1:9092 --topic TestTopic

6、offset

-- Get the consumer offsets for a topic
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper=localhost:2181 --topic=mytopic --group=my_consumer_group

7、Group

-- List the consumer groups known to Kafka
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list (old api)

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list (new api)

-- View the details of a consumer group
/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group

8、查看topic内容

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic trigger_web_log

/web/kafka/bin/kafka-console-consumer.sh -bootstrap-server 10.7.6.25:9092 --from-beginning --topic topic_adx_bid_win

-- Get number of messages in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

-- Get the earliest offset still in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

-- Get the latest offset still in a topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

-- Read from __consumer_offsets
Add the following property to config/consumer.properties: exclude.internal.topics=false

bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

-- Getting the last five message of a topic
kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e

-- 从指定offset查看某topic某partition

zhao@zhao:~/liujichao$ /web/kafka/bin/kafka-console-consumer.sh --bootstrap-server 10.7.6.25:9092 --topic topic_adx_up_request --partition 5 --offset 10364000

{"seId":"*******","rawOpId":"","opId":"517bc192b2c8b156","advId":"ff9b16c6-e30d-4891-9496-978dd8c1399d",。。。。}

9、Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

10、删除topic

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper hadoop242:2181,hadoop243:2181,hadoop244:2181  --topic test --delete

备注:以上操作逻辑删除

zhao@zhao:~/liujichao$ /usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper server-1:2181 --topic test  --delete

Topic test is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

11、两Kafka集群topic同步

    kafka-mirror-maker.sh,具体使用在V1.0中,这个功能是有问题,需要升级Kafka

12、查看指定消费组信息下面使用的topic offset 情况

/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 172.16.0.69:9092 --group group --describe

13、重置offset

    zookeeper中,设置set /consumers/console-consumer-29012/offsets/kafka_direct/1 0

14、根据主题查看各个分区的offset

zhao@zhao:~/liujichao$ /home/zhao/software/kafka_2.11-0.10.2.1/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list server-1:9092 --time -1 --topic topic_adx_dm_dsp
topic_adx_dm_dsp:2:67278
topic_adx_dm_dsp:1:67276
topic_adx_dm_dsp:0:67278
备注:time为-1时表示最大值,time为-2时表示最小值
  15、根据消费组查看各个分区的offset
/home/zhao/software/kafka_2.11-0.10.2.1/bin/kafka-consumer-groups.sh --bootstrap-server 10.238.221.110:9092,10.238.221.153:9092,10.238.221.124:9092 --describe --group TEST_GROUP_1

  16、


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: