【Kafka】Kafka安装、配置、使用

1. 安装Kafka

Kafka 官网https://kafka.apache.org/

上传文件至云服务器

解压Kafka

[root@VM-8-7-centos opt]# cd /opt
[root@VM-8-7-centos opt]# mkdir moudle
[root@VM-8-7-centos opt]# cd software/
[root@VM-8-7-centos software]# tar -zxvf kafka_2.12-3.0.0.tgz -C /opt/moudle/
1
2
3
4

修改名称

[root@VM-8-7-centos moudle]# mv kafka_2.12-3.0.0/ kafka

[root@VM-8-7-centos bin]# pwd
/opt/moudle/kafka/bin
1
2
3
4

修改配置文件

[root@VM-8-7-centos kafka]# cd config/
[root@VM-8-7-centos config]# ls
connect-console-sink.properties    connect-file-source.properties   consumer.properties  server.properties
connect-console-source.properties  connect-log4j.properties         kraft                tools-log4j.properties
connect-distributed.properties     connect-mirror-maker.properties  log4j.properties     trogdor.conf
connect-file-sink.properties       connect-standalone.properties    producer.properties  zookeeper.properties
[root@VM-8-7-centos config]# vim server.properties 
[root@VM-8-7-centos config]# pwd
/opt/moudle/kafka/config
1
2
3
4
5
6
7
8
9

修改地方1：broker.id

修改地方2：日志

修改地方3：Zookeeper

2. Kafka使用

2.0 集群分发脚本xsync(重要)

scp课程

scp安全拷贝
scp可以实现服务器与服务器之间的数据拷贝。（from server1 to server2）

2.0.1 scp命令

scp    -r        $pdir/$fname             $user@$host:$pdir/$fname
命令   递归       要拷贝的文件路径/名称        目的地用户@主机:目的地路径/名称
1
2

[root@VM-8-7-centos moudle]# scp -r kafka/ root@139.9.133.30:/opt/moudle
The authenticity of host '139.9.133.30 (139.9.133.30)' can't be established.
ECDSA key fingerprint is SHA256:CNtTiB8P/OCKir8Dx2nVGb461U8fuwHh9v3saUBMFKo.
ECDSA key fingerprint is MD5:6f:89:b0:b1:d8:55:9c:1d:82:1e:6e:54:ac:ca:ae:fb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '139.9.133.30' (ECDSA) to the list of known hosts.
root@139.9.133.30's password: 
javax.ws.rs-api-2.1.1.jar                                                                                            100%  124KB   3.5MB/s   00:00    
osgi-resource-locator-1.0.3.jar                                                                                      100%   19KB 996.1KB/s   00:00    
zstd-jni-1.5.0-2.jar                                                                                                 100% 6604KB 508.0KB/s   00:13    
reflections-0.9.12.jar                                                                                               100%  103KB 494.4KB/s   00:00    
jetty-servlet-9.4.43.v20210629.jar                                                                                   100%  142KB 547.7KB/s   00:00    
jakarta.annotation-api-1.3.5.jar                                                                                     100%   24KB 541.1KB/s   00:00   
1
2
3
4
5
6
7
8
9
10
11
12
13

2.0.2 rsync远程同步工具

rsync主要用于备份和镜像。具有速度快、避免复制相同内容和支持符号链接的优点。

rsync和scp区别：用rsync做文件的复制要比scp的速度快，rsync只对差异文件做更新。scp是把所有文件都复制过去。

rsync    -av       $pdir/$fname             $user@$host:$pdir/$fname
命令     选项参数    要拷贝的文件路径/名称       目的地用户@主机:目的地路径/名称
1
2

2.0.3 写一个集群分发脚本xsync (Shell 脚本)

#!/bin/bash

#1. 判断参数个数
if [ $# -lt 1 ]
then
    echo Not Enough Arguement!
    exit;
fi

#2. 遍历集群所有机器
for host in 139.9.133.30 14.116.254.124
do
    echo ====================  $host  ====================
    #3. 遍历所有目录，挨个发送

    for file in $@
    do
        #4. 判断文件是否存在
        if [ -e $file ]
            then
                #5. 获取父目录
                pdir=$(cd -P $(dirname $file); pwd)

                #6. 获取当前文件的名称
                fname=$(basename $file)
                ssh $host "mkdir -p $pdir"
                rsync -av $pdir/$fname $host:$pdir
            else
                echo $file does not exists!
        fi
    done
done


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

修改脚本 xsync 具有执行权限
[atguigu@hadoop102 bin]$ chmod +x xsync
1
2

2.1 Zookeeper安装

课程：https://www.bilibili.com/video/BV1to4y1C7gw?p=7&vd_source=240d9002f7c7e3da63cd9a975639409a

// todo

2.2 对Kafka进行分发

2.2.1 执行同步脚本

2.2.2 三台云主机配置Kafka环境变量

# jdk1.8
export JAVA_HOME=/usr/local/java/jdk1.8.0_131
export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin

# Nodejs
export PATH=/usr/local/node/node-v12.16.3-linux-x64/bin:$PATH

# maven
export MAVEN_HOME=/opt/maven/apache-maven-3.6.3
export PATH=$MAVEN_HOME/bin:$PATH


# kafka
export KAFKA_HOME=/opt/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin

[root@VM-8-7-centos moudle]# source /etc/profile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18