将自定义 Docker 日志转换为 Fluentd 到 Elastic Search

Question

我将发布一些包含通用日志框架（用 golang 编写）的 Docker 容器。日志格式是 JSON 格式。

此自定义 json 日志记录格式中有不同的数据，我希望使用 Kibana 对其进行索引/搜索。我的理解是我需要转换/过滤这些数据，但即使在 RTFM 之后我也很难理解这是如何完成的。我必须从 JSON 中提取 JSON？

Docker 日志中所示的最小示例应用程序的一些示例输出：

{"app_name":"SampleApp","app_port":6666,"app_version":"0.0.2","file":"/build/examples/sample/app/runmain/main.go:131","func":"example.com/code/microservices/examples/sample/app/runmain.mainErr.func1","fw_version":"v0.0.1","level":"info","msg":"listening","time":"2024-01-18T20:31:39.163970213+07:00"}

数据进入 Fluentd 并记录为：

2024-01-18 20:31:39.000000000 +0000 f905b090d278: {"container_id":"f905b090d278ec2cc2f1f912acdbf8787a0a1c91d8ab7b00ad84e9da20c8c147","container_name":"/fervent_jemison","source":"stdout","log":"{\"app_name\":\"SampleApp\",\"app_port\":6666,\"app_version\":\"0.0.2\",\"file\":\"/build/examples/sample/app/runmain/main.go:131\",\"func\":\"example.com/code/microservices/examples/sample/app/runmain.mainErr.func1\",\"fw_version\":\"v0.0.1\",\"level\":\"info\",\"msg\":\"listening\",\"time\":\"2024-01-18T20:31:39.163970213+07:00\"}\r"}

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter docker.**>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type json
  </parse>
</filter>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host es
    port 9200
    user elastic
    password elastic
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    <buffer> 
      flush_interval 1s
    </buffer>
  </store>

  <store>
    @type stdout
  </store>
</match>

最初，我会将其全部部署在本地一台主机上。

任何提示或进一步的指导将不胜感激。这对我来说是一个新世界。

Answer 1

您正在处理由forward指令发送和接收的日志，因此您不能简单地解析JSON。

<filter docker.**>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type json
  </parse>
</filter>

相反，您应该使用正则表达式。看起来您已经完成了记录器的工作，因此日志将很容易解析。

我正在测试，这个正则表达式规则应该适合你

{"app_name":"(?<app_name>[^"]*)","app_port":(?<app_port>[^"]*),"app_version":"(?<app_version>[^"]*)","file":"(?<file>[^"]*)","func":"(?<func>[^"]*)","fw_version":"(?<fw_version>[^"]*)","level":"(?<level>[^"]*)","msg":"(?<msg>[^"]*)","time":"(?<time>[^"]*)

正如你所说，你是 fluidd 的新手，我只能推荐你使用这 2 个网站来测试你的解析正则表达式

https:// Fluentular.herokuapp.com/parse?regexp=%7B%22app_name%22%3A%22%28%3F%3Capp_name%3E%5B%5E%22%5D*%29%22%2C% 22app_port%22%3A%28%3F%3Capp_port%3E%5B%5E%22%5D*%29%2C%22app_version%22%3A%22%28%3F%3Capp_version%3E%5B%5E%22%5D *%29%22%2C%22file%22%3A%22%28%3F%3Cfile%3E%5B%5E%22%5D*%29%22%2C%22func%22%3A%22%28%3F %3Cfunc%3E%5B%5E%22%5D*%29%22%2C%22fw_version%22%3A%22%28%3F%3Cfw_version%3E%5B%5E%22%5D*%29%22%2C %22level%22%3A%22%28%3F%3Clevel%3E%5B%5E%22%5D*%29%22%2C%22msg%22%3A%22%28%3F%3Cmsg%3E%5B% 5E%22%5D*%29%22%2C%22time%22%3A%22%28%3F%3Ctime%3E%5B%5E%22%5D*%29&输入=%7B%22app_name%22%3A%22SampleApp %22%2C%22app_port%22%3A6666%2C%22app_version%22%3A%220.0.2%22%2C%22file%22%3A%22%2Fbuild%2Fexamples%2Fsample%2Fapp%2Frunmain%2Fmain.go%3A131 %22%2C%22func%22%3A%22example.com%2Fcode%2Fmicroservices%2Fexamples%2Fsample%2Fapp%2Frunmain.mainErr.func1%22%2C%22fw_version%22%3A%22v0.0.1%22%2C%22level %22%3A%22info%22%2C%22msg%22%3A%22listening%22%2C%22time%22%3A%222024-01-18T20%3A31%3A39.163970213%2B07%3A00%22%7D&time_format= 和 https://regex101.com/