0%

ELK(ElasticSearch, Logstash, Kibana) 搭建实时日志分析平台

Posted on Disqus:

文中所有软件版本均为最新版本5.X，网上ELK 5.X 版本资料较少，所以搭建过程种碰到了不少坑，仅记录简略搭建日志，后续如果有时间会整理详细搭建记录。

安装ElasticSearch

1
2
3
4
5
# https://www.elastic.co/downloads/
# 安装 elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.1.tar.gz
tar xzf elasticsearch-5.2.1.tar.gz
cd elasticsearch-5.2.1/
  1. 修改配置 vi config/elasticsearch.yml 【可选】
1
2
3
4
5
6
7
cluster.name=my-application
node.name=node-1
path.data=/tmp/elasticsearch/data
path.logs=/tmp/elasticsearch/logs
# 0.0.0.0 则开启外网访问
network.host=0.0.0.0
http.port=9200
  1. 创建 path.xx 文件夹后，启动ElasticSearch
1
2
./bin/elasticsearch
netstat -tln

安装elasticsearch-head

1
2
3
4
5
# 安装 elasticsearch-head
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install
# npm install -g grunt --registry=https://registry.npm.taobao.org
  1. 修改 vi Gruntfile.js 加入hostname: ‘0.0.0.0’,【可选】
1
2
3
4
5
6
7
8
9
10
connect: {
  server: {
    options: {
      hostname: '0.0.0.0',
      port: 9100,
      base: '.',
      keepalive: true
    }
  }
}
  1. 修改vi config/elasticsearch.yml, 加入以下内容，否则可能导致连接不上elasticsearch
1
2
http.cors.enabled: true
http.cors.allow-origin: "*"
  1. 开启 9100 端口并启动
1
2
3
4
5
6
# 如未开放 9100 端口需先开放端口
iptables -I INPUT -p tcp --dport 9100 -j ACCEPT
service iptables save

grunt server &
# 访问 http://IP:9100/

安装Logstash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.1.tar.gz
tar xzf logstash-5.2.1.tar.gz
cd logstash-5.2.1
# 新建配置文件 logstash-simple.conf ,配置以下内容
input {  
  stdin { }
  syslog {# logback，logstash没有官方支持，配合jar包 net.logstash.logback
  	type => "test-logback"
    host => "127.0.0.1"
    port => 4562
    codec => "json"
  }
  log4j { # log4j
    type => "test-log4j"
    host => "127.0.0.1"
    port => 4560
   }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
#启动服务
./bin/logstash -f logstash-simple.conf

Java项目配置

  1. log4j配置以下内容
1
2
3
4
5
6
7
8
# appender socket
log4j.appender.socket=org.apache.log4j.net.SocketAppender
# 端口为logstash log4j 配置端口
log4j.appender.socket.Port=4560
log4j.appender.socket.RemoteHost=127.0.0.1
log4j.appender.socket.layout=org.apache.log4j.Patte10000rnLayout
log4j.appender.socket.layout.ConversionPattern=%d [%-5p] [%l] %m%n
log4j.appender.socket.ReconnectionDelay=100
  1. logback 使用以下配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!--pom.xml 添加 jar 包 -->
<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.8</version>
</dependency>

<!--logback.xml 添加 udp 配置 -->
<appender name="stash" class="net.logstash.logback.appender.LogstashSocketAppender">
  <host>127.0.0.1</host>
  <!-- 端口为logstash logback 配置端口 -->
  <port>4562</port>
  <customFields>{"appname":"my log"}</customFields>
</appender>

<appender-ref ref="stash" />

安装Kibana

1
2
3
4
5
6
7
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.0.0-linux-x86_64.tar.gz
tar xzf kibana-5.0.0-linux-x86_64.tar.gz
cd kibana-5.0.0-linux-x86_64
# 安装x-pack
bin/kibana-plugin install x-pack
./bin/kibana
# 访问 http://IP:5601

【可选】使用Redis做消息队列防止日志丢失

  1. 新建配置文件，写入 Redis
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
input {  
  stdin { }
  syslog {
    type => "test-logback"
    host => "127.0.0.1"
    port => 4562
    codec => "json"
  }
  log4j {
    type => "test-log4j"
    host => "127.0.0.1"
    port => 4560
   }
}
output {
  # elasticsearch { hosts => ["localhost:9200"] }
  redis {   # 表示写入到redis中
    host => "127.0.0.1"    # 指明redis服务器地址
    port => 6379   # 指明redis端口
    data_type => "list"       # 指定数据类型为list
    key => "logstash"  # 指定存入的键
  }
  # stdout { codec => rubydebug }
}
  1. 从Redis 写入 ElasticSearch
1
2
3
4
5
6
7
8
9
10
11
12
13
input {  
  stdin { }
  redis {   # 表示从redis中取出数据
    host => "127.0.0.1"  
    port => 6379
    data_type => "list"
    key => "logstash"   # 指定去logstash这个键中取出数据
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }#写入 elasticsearch
  # stdout { codec => rubydebug }
}

【可选】使用Kafka做消息队列防止日志丢失

  1. 新建配置文件，写入 Kafka
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
input {  
  stdin { }
  syslog {
    type => "test-logback"
    host => "127.0.0.1"
    port => 4562
    codec => "json"
  }
  log4j {
    type => "test-log4j"
    host => "127.0.0.1"
    port => 4560
   }
}
output {
  # elasticsearch { hosts => ["localhost:9200"] }
  kafka {  #输出到kafka
    bootstrap_servers => "localhost:9092"  #生产者
    codec => plain {
        format => "%{host} %{logger_name} %{level} %{message}"
    }
  	compression_type => "snappy" # 压缩类型
    topic_id => "logstash"  #这个将作为主题的名称，将会自动创建
  }
  # stdout { codec => rubydebug }
}
  1. 从Kafka写入 ElasticSearch
1
2
3
4
5
6
7
8
9
10
11
12
input {  
  stdin { }
  kafka {
      bootstrap_servers => "localhost:9092"  #消费者们
      compression_type => "snappy" # 压缩类型
      consumer_threads => 5 # 线程 默认 1
  }
}
output {
  elasticsearch { hosts => ["localhost:9200"] }#写入 elasticsearch
  # stdout { codec => rubydebug }
}

ELK 集群搭建

ElasticSearch 集群搭建

  1. 修改 elasticsearch.yml (master)
    1
    2
    3
    # 同一局域网下elasticsearch会自动发现相关节点，所以不用配置 discovery.zen.xxx 等参数
    cluster.name: my-es-cluster
    node.name: node-1
  2. 另一台服务器修改 elasticsearch.yml (node)，如果是同一台服务器搭建集群，复制 elasticsearch 目录时注意清空data目录
    1
    2
    cluster.name: my-es-cluster
    node.name: node-2
  3. logstash 配置 hosts 数组即可
    1
    2
    3
    4
    output {
      elasticsearch { hosts => ["localhost:9200", "localhost:9201", "localhost:9202"] }
      stdout { codec => rubydebug }
    }

    Kafka 集群搭建

  4. 修改 server.properties
    1
    2
    3
    4
    # kafka 集群主要关注下面这两个参数极客
    broker.id=0 # 当前机器在集群中的唯一标识，和zookeeper的myid性质一样
    zookeeper.connect=localhost:2181,localhost:2182
    # windows 启动：.\bin\windows\kafka-server-start.bat .\config\server.properties
  5. logstash output 配置
    1
    2
    3
    4
    5
    6
    kafka {  #输出到kafka
      bootstrap_servers => "localhost:9092,localhost:9093,localhost:9094"  #生产者 logstash 5.X kafka 插件不在依赖zookeeper，所以去掉了 zk_connect
      compression_type => "snappy" # 压缩类型
      enable_metric => true
      topic_id => "logstash"  #这个将作为主题的名称，将会自动创建
    }
  6. logstash input 配置
    1
    2
    3
    4
    kafka {
        bootstrap_servers => "localhost:9092,localhost:9093,localhost:9094"  #消费者们
        consumer_threads => 5
    }

ElasticSearch 启动错误解决方案

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1. ElasticSearch默认内存2G，启动报以下错误，则要在 vi jvm.options 修改 jvm 内存 -Xms512m  
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)

2. ElasticSearch默认不允许root权限运行，报如下错误则要新建用户运行
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root

groupadd elsearch
useradd elsearch -g elsearch -p elasticsearch
usermod -g root elsearch
su elsearch

3. java.io.IOException: Permission denied
chown -R elsearch /tmp/elasticsearch

4. initial heap size [16777216] not equal to maximum heap size [536870912]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
sudo sysctl -w vm.max_map_count=262144

5.   failed to send join request to master [{node-1} ... {127.0.0.1}{127.0.0.1:9300} with the same id but is a different node instance];
集群搭建时出现该错误，解决方法为删除清空es目录下data目录

到这里ELK基本配置已经完了，但搭建中还遇到了几点问题没有解决，比如 logstash 怎么做集群？目前的方法是每台 server log 配置中都要配置需要连接的logstash地址，但这种在分布式视同中是不太合适的。