Logstash
部署 logstash
2.1）logstash 环境准备及安装
Logstash 是一个开源的数据收集引擎，可以水平伸缩，而且 logstash 整个 ELK当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地
https://github.com/elastic/logstash
https://baike.baidu.com/item/Ruby/11419 #基于 ruby 开发
2.1.1）环境准备
-
关闭防火墙和 selinux，并且安装 java 环境
[root@logstash ~]# systemctl stop firewalld
[root@logstash ~]# systemctl disable firewalld
[root@logstash ~]# setenforce 0
[root@logstash ~]# sed -i '7s#enforcing#disabled#g' /etc/selinux/config
[root@logstash ~]# yum -y install java-1.8.0-openjdk.x86_64
[root@logstash ~]# java -version
openjdk version "1.8.0_382"
OpenJDK Runtime Environment (build 1.8.0_382-b05)
OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)
2.1.2）安装 logstash
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.12.1/
[root@logstash ~]# cd /usr/local/src/
`上传rpm包到centos上面`
[root@logstash src]# ll
total 361028
-rw-r--r-- 1 root root 369691684 Nov 11 18:08 logstash-7.12.1-x86_64.rpm
[root@logstash src]# yum -y localinstall logstash-7.12.1-x86_64.rpm
[root@logstash src]# chown logstash.logstash /usr/share/logstash/data/ -R
2.2.3）测试输出到终端
https://www.elastic.co/guide/en/logstash/7.12/plugins-inputs-elasticsearch.html
[root@logstash src]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }' #标准输入和输出
hello
{
"host" => "logstash",
"message" => "hello",
"@version" => "1",
"@timestamp" => 2023-11-11T10:33:41.668Z
}
2.2.4）测试输出到文件
https://www.elastic.co/guide/en/logstash/7.12/plugins-inputs-elasticsearch.html
[root@logstash src]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file { path => "/tmp/log-%{+YYYY.MM.dd}messages.gz"} }'
[root@logstash src]# ll /tmp/log-2023.11.11messages.gz
-rw-r--r-- 1 root root 99 Nov 11 18:35 /tmp/log-2023.11.11messages.gz
2.2.5）测试输出到 elasticsearch
https://www.elastic.co/guide/en/logstash/7.12/plugins-inputs-elasticsearch.html
[root@logstash ~]# cat /etc/logstash/conf.d/test.conf
input {
stdin{}
}
output {
#stdout{}
elasticsearch {
hosts => ["192.168.139.31:9200","192.168.139.32:9200","192.168.139.33:9200"]
index => "stang-test-%{+YYYY.MM.dd}"
}
}
`测试文件是否写对`
[root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t
`将文件写入到elasticsearch`
[root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf
...........
version1
version2
version3
2.2.6）在elasticsearsh的head插件查看写入情况
2.2.7）logstash收集centos的系统日志到elasticsearsh
https://www.elastic.co/guide/en/logstash/7.12/plugins-inputs-file.html#plugins-inputs-file-path
`修改service文件以root启动，也可以用普通用户启动但是要收集系统日志需要修改权限比较麻烦`
[root@logstash ~]# vim /etc/systemd/system/logstash.service
.....
User=root
Group=root
.....
`修改配置`
[root@logstash ~]# cat /etc/logstash/conf.d/test.conf
input {
#stdin{}
file {
path => "/var/log/vmware-network*.log"
type => "vmware-network-log"
start_position => "beginning"
stat_interval => "3"
}
}
output {
#stdout{}
elasticsearch {
hosts => ["192.168.139.31:9200","192.168.139.32:9200","192.168.139.33:9200"]
index => "stang-sys%{+YYYY.MM.dd}"
}
}
`检查配置是否正常`
[root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t
`启动服务`
[root@logstash ~]# systemctl start logstash.service
`查看日志是否启动正常`
[root@logstash ~]# tail -f /var/log/logstash/logstash-plain.log
`查看启动的端口`
[root@logstash ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*
LISTEN 0 50 [::ffff:127.0.0.1]:9600 [::]:*
`追加内容到文件`
[root@logstash ~]# echo v1 >> /var/log/vmware-network.1.log
`es的head插件查看es的变化`
