Logstash无法理解的内存不足错误

Question

我有一个Logstash 7.6.2 docker，由于内存泄漏而停止运行。每次执行管道后，Logstash似乎都不会释放内存。我应该怎么做才能确定问题的根源？我该如何解决？

任何帮助都是欢迎^^。

这是我在日志中看到的错误。这些只是Traceback的前5行。我将其余部分上传到我的github there中的文件中。

logstash    | [2020-04-08T18:15:42,960][INFO ][logstash.outputs.file    ][rawweb] Closing file /output/web_data.json
 logstash    | [2020-04-08T18:15:43,353][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space
 logstash    | [2020-04-08T18:15:43,367][ERROR][org.logstash.execution.WorkerLoop][rawclient] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
 logstash    | org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `pop' for nil:NilClass
 logstash    |   at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.awesome_print_minus_1_dot_7_dot_0.lib.awesome_print.inspector.awesome(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/inspector.rb:117) ~[?:?]

这里是我用来配置Logstash Docker的docker-compose.yml”>

version: '2.4'    
services:
      logstash:
        image: docker.elastic.co/logstash/logstash:7.6.2
        container_name: logstash
        environment:
          LS_JAVA_OPTS: "-Xmx7g -Xms4g"
          REQUEST_FREQUENCY: 600 #seconds
        volumes:
          - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
          - ./logstash/pipelines.yml:/usr/share/logstash/config/pipelines.yml
          - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
          - ./logstash/tests:/testscripts:ro
          - /root/logstash_output/:/output/
        ports:
          - "9600:9600"
        mem_limit: 7000M
        mem_reservation: 100M
我的pipelines.yml

文件

- pipeline.id: rawclient
  path.config: "/usr/share/logstash/pipeline/logclient.conf"
  pipeline.batch.size: 10000000

 - pipeline.id: rawweb
   path.config: "/usr/share/logstash/pipeline/logweb.conf"
   pipeline.batch.size: 10000000
我的。conf

个文件之一。基本上，它执行包含curl请求的。sh脚本。该请求的结果是管道的输入。进行治疗。然后将结果存储在文件中。这两个管道执行相同的操作，唯一的不同是发出的curl请求。

input {
  exec {
    command => "bash /testscripts/logclient_1.sh"
    codec => "json"
    interval => "600"
  }
}

filter {
    mutate {
        rename => ["connection/start_time", "start_time" ]
        rename => ["connection/end_time", "end_time" ]
        rename => ["connection/duration", "duration" ]
        rename => ["connection/destination_ip_address", "destination_ip_address" ]
        rename => ["connection/status", "status" ]
        rename => ["device/last_ip_address", "last_ip_address" ]
        rename => ["user/sid", "sid" ]
        # rename => ["binary/application_category", "application_category" ]
        rename => ["binary/application_name", "application_name" ]
        rename => ["binary/executable_name", "executable_name" ]
        remove_field => ["@timestamp"]
        remove_field => ["@version"]
        add_field => { "connection_type" => "client" }
    }
}
output {
  file {
   path => "/output/client_data.json"
   codec => "json"
 }
  stdout {
   codec => rubydebug
 }
}
我的logstash.yml

文件

http.host: "0.0.0.0"
xpack.monitoring.enabled: false
Thanks for all the help :slightly_smiling_face:

我有一个Logstash 7.6.2泊坞窗，由于内存泄漏而停止运行。每次执行管道后，Logstash似乎都不会释放内存。我应该怎么做才能识别...

Answer 1

您的管道批次大小为巨大