kafka集群搭建_kafka3.4.0集群搭建

1、准备三台服务，ip如下

192.168.37.183
192.168.37.150
192.168.37.86

2、防火墙
kafka服务应限制访问ip，将集群节点和需要访问kafka的ip添加到trusted区域
安装、启动防火墙

yum install firewalld.noarch
systemctl enable firewalld.service
systemctl start firewalld.service

不同的服务器执行不同的命令
（1）192.168.37.183服务器

firewall-offline-cmd --zone=trusted --add-source=192.168.37.150
firewall-offline-cmd --zone=trusted --add-source=192.168.37.86
systemctl reload firewalld
firewall-cmd --list-all-zones

（2）192.168.37.150服务器

firewall-offline-cmd --zone=trusted --add-source=192.168.37.183
firewall-offline-cmd --zone=trusted --add-source=192.168.37.86
systemctl reload firewalld
firewall-cmd --list-all-zones

（3）192.168.37.86服务器

firewall-offline-cmd --zone=trusted --add-source=192.168.37.183
firewall-offline-cmd --zone=trusted --add-source=192.168.37.150
systemctl reload firewalld
firewall-cmd --list-all-zones

3、kafka集群搭建
新建kafka日志目录

mkdir /home/trimps/kafka-logs

将kafka_2.13-2.7.2.tgz文件上传到/home/trimps/software目录，解压

tar -zxvf kafka_2.13-2.7.2.tgz

主要配置（各服务器配置不同）
（1）192.168.37.183服务器

vi server.properties
log.dirs=/home/trimps/kafka-logs
zookeeper.connect=192.168.37.183:3292,192.168.37.150:3292,192.168.37.86:3292
log.retention.hours=168
log.retention.bytes=107374182400
broker.id=0
listeners=PLAINTEXT://192.168.37.183:9092

（2）192.168.37.150服务器

vi server.properties
log.dirs=/home/trimps/kafka-logs
zookeeper.connect=192.168.37.183:3292,192.168.37.150:3292,192.168.37.86:3292
log.retention.hours=168
log.retention.bytes=107374182400
broker.id=1
listeners=PLAINTEXT://192.168.37.150:9092

（3）192.168.37.86服务器

vi server.properties
log.dirs=/home/trimps/kafka-logs
zookeeper.connect=192.168.37.183:3292,192.168.37.150:3292,192.168.37.86:3292
log.retention.hours=168
log.retention.bytes=107374182400
broker.id=2
listeners=PLAINTEXT://192.168.37.86:9092

启动、停止命令

bin/kafka-server-start.sh -daemon config/server.properties
bin/kafka-server-stop.sh -daemon config/server.properties

4、测试

启动生产者发送消息、启动消费者接收消息
（1）192.168.37.183服务器

[root@0003 kafka_2.13-2.7.2]# bin/kafka-topics.sh --bootstrap-server 192.168.37.183:9092 --create --topic topic_test --partitions 3 --replication-factor 2
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic topic_test.
 
[root@0003 kafka_2.13-2.7.2]# bin/kafka-topics.sh --bootstrap-server 192.168.37.183:9092 --describe --topic topic_test
Topic: topic_test       PartitionCount: 3       ReplicationFactor: 2    Configs: segment.bytes=1073741824,retention.bytes=107374182400
        Topic: topic_test       Partition: 0    Leader: 2       Replicas: 2,0   Isr: 2,0
        Topic: topic_test       Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: topic_test       Partition: 2    Leader: 0       Replicas: 0,1   Isr: 0,1
 
[root@0003 kafka_2.13-2.7.2]# bin/kafka-console-producer.sh --broker-list 192.168.37.183:9092 --topic topic_test
>qaq
>

（2）192.168.37.150服务器

[root@0002 kafka_2.13-2.7.2]# bin/kafka-topics.sh --bootstrap-server 192.168.37.150:9092 --describe --topic topic_test
Topic: topic_test       PartitionCount: 3       ReplicationFactor: 2    Configs: segment.bytes=1073741824,retention.bytes=107374182400
        Topic: topic_test       Partition: 0    Leader: 2       Replicas: 2,0   Isr: 2,0
        Topic: topic_test       Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: topic_test       Partition: 2    Leader: 0       Replicas: 0,1   Isr: 0,1
 
[root@0002 kafka_2.13-2.7.2]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.37.150:9092 --from-beginning --topic topic_test
exit
world
hello
quit
qaq

（3）192.168.37.86服务器

[root@0001 kafka_2.13-2.7.2]# bin/kafka-topics.sh --bootstrap-server 192.168.37.86:9092 --describe --topic topic_test
Topic: topic_test       PartitionCount: 3       ReplicationFactor: 2    Configs: segment.bytes=1073741824,retention.bytes=107374182400
        Topic: topic_test       Partition: 0    Leader: 2       Replicas: 2,0   Isr: 2,0
        Topic: topic_test       Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: topic_test       Partition: 2    Leader: 0       Replicas: 0,1   Isr: 0,1
 
[root@0001 kafka_2.13-2.7.2]# bin/kafka-console-consumer.sh --bootstrap-server 192.168.37.86:9092 --from-beginning --topic topic_test
exit
world
hello
quit
qaq