使用docker-compose快速搭建ELK_elk docker-compose

ubuntu系统

查看Elasticsearch版本

docker search Elasticsearch
1

拉取镜像(7.1版本)

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.1.0
1

新建docker-compose.yml 文件

version: '2.2'
services:
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:7.1.0
    container_name: kibana7
    environment:
      - I18N_LOCALE=zh-CN
      - XPACK_GRAPH_ENABLED=true
      - TIMELION_ENABLED=true
      - XPACK_MONITORING_COLLECTION_ENABLED="true"
    ports:
      - "5601:5601"
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    container_name: es7_01
    environment:
      - cluster.name=xttblog
      - node.name=es7_01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.seed_hosts=es7_01
      - cluster.initial_master_nodes=es7_01,es7_02
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es7data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    container_name: es7_02
    environment:
      - cluster.name=xttblog
      - node.name=es7_02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.seed_hosts=es7_01
      - cluster.initial_master_nodes=es7_01,es7_02
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es7data2:/usr/share/elasticsearch/data
volumes:
  es7data1:
    driver: local
  es7data2:
    driver: local
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

运行docker中的compose文件(在上面yml文件的目录运行)

• docker-compose up -d : 在后台运行, 不打印日志

加大系统运行内存

• 如果报错, “max virtual memory areas vm.max_map_count [65530] is too low, increase to at least”那说明你设置的 max_map_count 小了
• 编辑 /etc/sysctl.conf
• 追加以下内容：vm.max_map_count=262144保存后
• 重新启动：sysctl -p

调整elasticsearch的jvm内存(额外操作,可以不加)

• [root@localhost /]# find / -name jvm.options

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g      #改成512m
-Xmx1g      #改成512m

################################################################
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

在浏览器去登录ES与kibana与cerebro

• 5601登录kibana

• 9200登录ES

• 9000登录cerebro

Linux安装logstash

进入到elasticsearch官网下载和elasticsearch同版本的logstash

将logstash上传到服务器安装

解压logstash:

  tar -zxvf logstash-7.1.0.tar.gz
1

顺便安装一个jdk1.8(安装好的可以跳过)

apt-get install openjdk-8-jdk
1

配置logstash的配置文件

下载测试数据集

• 测试数据集, 就是在我们启动logstash的时候往ES中导入数据, 供我们测试
• 网站：https://grouplens.org/datasets/movielens/

导入数据到elasticsearch

• 传入需要测试的数据集到/opt/elk文件夹下：

• 数据集文件里面的movies.csv就是我们需要导入的数据

建立配置文件

在logstash的bin目录下新建配置文件

input {
  file {
    path => "/opt/elk/ml-latest-small/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|"}
    remove_field => ["path", "host", "@timestamp","message"]
  }

  mutate {
    split => ["content", "("]
    add_field => {"title" => "%{[content][0]}"}
    add_field => {"year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host", "@timestamp","message","content"]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.8.109:9200"]
    index => "movies"
    document_id => "%{id}"
  }
  stdout {}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

启动logstash

在logstash 的bin目录下启动

cd /opt/elk/logstash-7.1.0/bin && ./logstash -f logstash.conf
1

通过日志，我们可以看到数据被导入到elasticsearch中，我们同样可以在kibana中看到数据已经被导入elasticsearch。

微信公众号：海哥python