elk性能问题

elk性能问题

各位大佬请教个问题

如上图示，我用filebeat收集本地log日志到elsearch中，中间经过了logstash做了格式处理，所有服务器都是虚拟化平台里安装的独立服务器。

但目前发现效率非常低，从filebeat上看，速度还没达到1M/s的上传速度。现在不知道瓶颈在哪里？

下面是我的相关配置文件

1、filebeat的filebeat.yml配置

filebeat.inputs:

  • type: log

    enabled: true

    paths:

    • /log/*.log

      output.logstash:

      hosts: ["192.168.50.21:9600","192.168.50.22:9600","192.168.50.23:9600"]

      loadbalance: true

2、logstash的logstash.conf（其中一个）

input {

beats {

port => "9600"

}

}

filter {

mutate {

split => {"message" => " "}

}

mutate {

add_field => {

"day0" => "%{[message][0]}"

"time" => "%{[message][6]}"

"local_ip" => "%{[message][11]}"

"remote_ip" => "%{[message][13]}"

}

}

mutate {

convert => {

"time" => "string"

"local_ip" => "string"

"remote_ip" => "string"

}

}

date {

match => ["day0", "yyyy-MM-dd"]

add_field => ["day", "%{day0}"]

}

mutate {

remove_field => ["message","day0","host","@timestamp","@version","path","input.type","agent.type","input","type","agent","log.file.path","log","file","ecs.version","ecs","version","log.offset","offset"]

remove_field => ["log.offset","log.file.path"]

}

}

output {

elasticsearch {

hosts => [ "192.168.50.11:9200","192.168.50.12:9200","192.168.50.13:9200"]

index => "natlog-1"

}

}

3、elsearch的elasticsearch.yml配置（其中一个）

cluster.initial_master_nodes: ["es-itcast-cluster"]

node.name: node01

node.master: true

node.data: true

discovery.zen.minimum_master_nodes: 2

bootstrap.memory_lock: false

bootstrap.system_call_filter: false

network.host: 0.0.0.0

discovery.seed_hosts: ["192.168.50.11","192.168.50.12","192.168.50.13"]

http.cors.enabled: true

http.cors.allow-origin: ""

http.cors.allow-methods: "GET"

回答

logstash那个服务器你看下配置，然后对应调整logstash里面 pipeline.workers pipeline.batch.size ,调大点。

以上是 elk性能问题 的全部内容， 来源链接： utcz.com/a/79680.html

回到顶部