在k8s中安装kafka单机、集群说明_kubenetes kafka 单机

作者：在线问答5 | 2024-07-22 08:14:15

踩

kubenetes kafka 单机

一、安装单节点kafka（自己测试过）

1、创建zookeeper服务

zookeeper-service.yaml内容如下：

#Service
apiVersion: v1
kind: Service
metadata:
    name: kafka-zookeeper-service
    namespace: paas-basic
    labels:
        name: zookeeper-service
spec:    
    selector:
        name: kafka-zookeeper-pod
    sessionAffinity: ClientIP
    type: NodePort
    ports:
    - name: "zookeeper"
      port: 2181
      targetPort: 2181

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

zookeeper-deploy.yaml内容如下：

#Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
    name: kafka-zookeeper-deploy
    namespace: paas-basic
    labels:
        name: zookeeper-deploy-label
spec:
    replicas: 1
    selector:
      matchLabels:
        name: kafka-zookeeper-pod
    template:
        metadata:
            labels:
                name: kafka-zookeeper-pod
        spec:
            terminationGracePeriodSeconds: 30  #k8s正确、优雅地关闭应用，等待时间30秒
            nodeSelector:
              kafka: "true"
            containers:
            - name: "kafka-zookeeper"
              image: wurstmeister/zookeeper
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 2181
              volumeMounts:
              - name: zk-data
                readOnly: false
                mountPath: /opt/zookeeper-3.4.13/data
            volumes:
            - name: zk-data
              hostPath:
                path: /home/k8s-1.19.2/paas-basic/kafka/zookeeper_data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

2、创建kafka。

kafka-service.yaml的内容如下：

#Service
apiVersion: v1
kind: Service
metadata:
    name: kafka-service
    namespace: paas-basic
    labels:
        name: kafka-service
spec:    
    selector:
        name: kafka-pod
    sessionAffinity: ClientIP
    type: NodePort
    ports:
    - name: "kafka"
      port: 9092
      targetPort: 9092
      nodePort: 30092

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

kafka-deploy.yaml的内容如下：

#Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
    name: kafka-deploy
    namespace: paas-basic
    labels:
        name: kafka-deploy
spec:
    replicas: 1
    selector: 
      matchLabels:
        name: kafka-pod
    template:
        metadata:
            labels:
                name: kafka-pod
        spec:
            terminationGracePeriodSeconds: 30  #k8s正确、优雅地关闭应用，等待时间30秒
            nodeSelector:
              kafka: "true"
            hostname: kafka-hostname       #设置pod的hostaname
            containers:
            - name: "kafka"
              image: wurstmeister/kafka:2.12-2.3.0
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9092
              volumeMounts:
              - name: kafka-volume
                mountPath: /kafka
              env:
              - name: KAFKA_ADVERTISED_PORT
                value: "30092"
              - name: KAFKA_MESSAGE_MAX_BYTES
                value: "1073741824"
              - name: KAFKA_REPLICA_FETCH_MAX_BYTES
                value: "1073741824"
              - name: KAFKA_BATCH_SIZE
                value: "4096"
              - name: KAFKA_ADVERTISED_HOST_NAME
                value: "192.168.180.37"
              - name: KAFKA_ZOOKEEPER_CONNECT
                value: kafka-zookeeper-service.paas-basic:2181
              - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
                value: "true"
              - name: KAFKA_LOG_RETENTION_HOURS
                value: "24"
              - name: LOG_CLEANUP_POLICY
                value: "delete"
            volumes:
            - name: kafka-volume
              hostPath:
                path: /home/k8s-1.19.2/paas-basic/kafka/volume
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

3、创建manager

manager-service.yaml

#Service
apiVersion: v1
kind: Service
metadata:
    name: kafka-manager
    namespace: paas-basic
    labels:
        name: manager-service
spec:    
    selector:
        name: kafka-manager-pod
    sessionAffinity: ClientIP
    type: NodePort
    ports:
    - name: "manager"
      port: 9000
      targetPort: 9000
      nodePort: 30900
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

manager-deploy.yaml

#Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
    name: kafka-manager-deploy
    namespace: paas-basic
    labels:
        name: manager-deploy
spec:
    replicas: 1
    selector:
      matchLabels:
        name: kafka-manager-pod
    template:
        metadata:
            labels:
                name: kafka-manager-pod
        spec:
            terminationGracePeriodSeconds: 1  #k8s正确、优雅地关闭应用，等待时间30秒
            nodeSelector:
              kafka: "true"
            containers:
            - name: "kafka-manager"
              image:  sheepkiller/kafka-manager
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 9000
              env:
              - name: ZK_HOSTS
                value: kafka-zookeeper-service.paas-basic:2181
              #- name: KAFKA_ZOOKEEPER_CONNECT
                #value: kafka-zookeeper-service.paas-basic:2181

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

二、创建多节点的kafka集群（未测试）

下面将搭建3个节点的kafka集群。这里采用了3个Deployment来运行Kafka和Zookeeper，其实更优雅的方式是使用StatefulSet。Kubernetes的官方文档上有使用StatefulSet搭建Zookeeper集群的范例。但是使用StatefulSet搭建Zookeeper和Kafka时，Zookeeper的myid和Kafka的brokerID就不能预先设置了，因此需要在镜像构建过程中加入相关的操作，而Docker Hub中的绝大多数镜像都不包含这一逻辑。而Deployment虽然不够优雅，但是可以对各节点预先配置，运行起来相对简单，可以说各有所长。

1）搭建zookeeper集群
首先创建zookeeper的yaml文件。

zookeeper-svc2.yaml的内容如下：

apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo2
  labels:
    app: zookeeper-2
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-2
---
apiVersion: v1
kind: Service
metadata:
  name: zoo3
  labels:
    app: zookeeper-3
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

zookeeper-deployment2.yaml的内容如下：

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-1
      name: zookeeper-1
  template:
    metadata:
      labels:
        app: zookeeper-1
        name: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-2
      name: zookeeper-2
  template:
    metadata:
      labels:
        app: zookeeper-2
        name: zookeeper-2
    spec:
      containers:
      - name: zoo2
        image: digitalwonderland/zookeeper
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "2"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-3
      name: zookeeper-3
  template:
    metadata:
      labels:
        app: zookeeper-3
        name: zookeeper-3
    spec:
      containers:
      - name: zoo3
        image: digitalwonderland/zookeeper
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "3"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: ZOOKEEPER_SERVER_2
          value: zoo2
        - name: ZOOKEEPER_SERVER_3
          value: zoo3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

分别执行kubectl apply -f zookeeper-svc2.yaml 和 kubectl apply -f zookeeper-deployment2.yaml

这里创建了3个deployment和3个service，一一对应。这样，三个实例都可以对外提供服务。创建完成后，需要用kubectl logs查看一下三个Zookeeper的pod的日志，确保没有错误发生，并且在3个节点的日志中，有类似下面的语句，则表明Zookeeper集群已顺利搭建成功。

2019-06-24 05:22:06,582 [myid:3] - INFO [QuorumPeer[myid=3]/0.0.0.0:2181:Leader@371] - LEADING - LEADER ELECTION TOOK - 14641

2）搭建Kafka集群
同样创建3个deployment和3个service，编写kafka-svc2.yaml和kafka-deployment2.yaml如下：

kafka-svc2.yaml配置如下

apiVersion: v1
kind: Service
metadata:
  name: kafka-service-1
  labels:
    app: kafka-service-1
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-service-1
    targetPort: 9092
    nodePort: 30901
    protocol: TCP
  selector:
    app: kafka-service-1
---

apiVersion: v1
kind: Service
metadata:
  name: kafka-service-2
  labels:
    app: kafka-service-2
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-service-2
    targetPort: 9092
    nodePort: 30902
    protocol: TCP
  selector:
    app: kafka-service-2
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service-3
  labels:
    app: kafka-service-3
spec:
  type: NodePort
  ports:
  - port: 9092
    name: kafka-service-3
    targetPort: 9092
    nodePort: 30903
    protocol: TCP
  selector:
    app: kafka-service-3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

kafka-deployment2.yaml 配置如下，注意需要将[kafka-service2的clusterIP]等三项替换成实际的clusterIP。

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-service-1
  template:
    metadata:
      labels:
        name: kafka-service-1
        app: kafka-service-1
    spec:
      containers:
      - name: kafka-1
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: [kafka-service1的clusterIP]
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181,zoo2:2181,zoo3:2181
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_CREATE_TOPICS
          value: mytopic:2:1
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-deployment-2
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-service-2
  template:
    metadata:
      labels:
        name: kafka-service-2
        app: kafka-service-2
    spec:
      containers:
      - name: kafka-2
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: [kafka-service2的clusterIP]
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181,zoo2:2181,zoo3:2181
        - name: KAFKA_BROKER_ID
          value: "2"
---

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-deployment-3
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kafka-service-3
  template:
    metadata:
      labels:
        name: kafka-service-3
        app: kafka-service-3
    spec:
      containers:
      - name: kafka-3
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: [kafka-service3的clusterIP]
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181,zoo2:2181,zoo3:2181
        - name: KAFKA_BROKER_ID
          value: "3"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

3）集群功能测试
测试方法基本同单集群的情况，这里就不赘述了。不同的是，这次可以将不同的节点作为生产者和消费者。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/在线问答5/article/detail/864157