FluentD 中如何仅发送满足某些条件的记录

Question

有什么问题吗？

我想根据条件向 ElasticSearch 发送一些记录。

这是我的主要会议。

<source>
  bind 0.0.0.0
  <parse>
    expression /<SOME CUSTOM REGEXP>/
    @type regexp
  </parse>
  port 5514
  tag main_tag
  <transport tcp>
  </transport>
  @type syslog
</source>

这是我发送到 local3 的 match。

<match main_tag.local3.*>
      @type stdout
      @id debug_output
</match>

结果在 /var/log/ Fluent/fluidd.log

2024-12-11 19:34:57.576728732 +0000 main_tag.local3.info: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"} 

我只想发送一些记录到 ElasticSearch....

例如仅使用 "as" 值 NOT 200。

或使用 "x" 值 apache OR nginx

或者我必须考虑的其他一些条件。

或者根据情况发送到不同的地方。

简而言之，如果可能的话，如何管理条件。

环境

- Fluentd version: fluentd 1.17.0 (e763c0761c44d9734b6aa374371387a2e8406522)
- Fluent Package version: fluent-package 5.1.0
- Operating system: Ubuntu 24.04.1 LTS
- Kernel version: 6.8.0-1019-aws

Answer 1

您可以尝试

rewrite_tag_filter

插件，根据您的情况配置规则。首先，您需要将 解析 系统日志源输入为

json

。

这是一个基于您的一些要求的流畅配置示例，我使用

dummy

输入进行了测试。让我知道你的想法！！.

<source>
  @type dummy
  dummy [
    {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"nginx","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"},
    {"h":"10.0.0.205","x":"other","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
  ]
  tag main_tag
</source>

<match main_tag>
  @type rewrite_tag_filter
  <rule>
    key as
    pattern /^200$/
    tag 200.${tag}
  </rule>
   <rule>
    key x
    pattern /^nginx|apache$/
    tag server.${tag}
  </rule>
  <rule>
    key message
    pattern /.+/
    invert true
    tag unmatched.${tag}
  </rule>
  # more rules
</match>

<match 200.main_tag>
  @type stdout
</match>

<match server.main_tag>
  @type stdout
</match>

<match unmatched.main_tag>
  @type stdout
</match>

这应该生成以下输出，其中记录与相应标签匹配。

2024-12-14 03:35:33 +0000 [info]: #0 starting fluentd worker pid=17 ppid=7 worker=0
2024-12-14 03:35:33 +0000 [info]: #0 fluentd worker is now running worker=0
2024-12-14 03:35:34.079746354 +0000 200.main_tag: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:35.082569552 +0000 server.main_tag: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:36.084676729 +0000 server.main_tag: {"h":"10.0.0.205","x":"nginx","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}
2024-12-14 03:35:37.086780304 +0000 unmatched.main_tag: {"h":"10.0.0.205","x":"other","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"400","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}