Kubernetes 中流畅的位分割多行 JSON 日志（带有 MuleSoft RTF 的 AKS）

问题描述 投票：0回答：1

我正在使用 Fluent Bit 解析来自部署在 Azure Kubernetes 服务 (AKS) 群集中的 MuleSoft Runtime Fabric (RTF) 的日志。我的应用程序生成的日志有一个标头，后跟一些元数据（例如线程名称和日志级别），然后是 JSON 负载。然而，Fluent Bit 通过换行符分割 JSON 有效负载 ( ），导致 JSON 的每一行在 Elasticsearch 中显示为单独的日志条目。

标准输出日志

[2024-12-09 16:58:07.453] INFO  LoggerMessageProcessor [[MuleRuntime].uber.05: [slb-rtf-aks-new].slb-rtf-aksFlow.CPU_INTENSIVE @4d2de1ee] [event: c33528c0-b64e-11ef-be04-0215ddf489fc]: {
  "status": "alive"
}

流畅位日志中的条目

[2024/12/09 11:36:08] [debug] [input:tail:tail.0] excluded=/var/log/containers/mule-clusterip-service-5fb85849cf-274vg_rtf_mule-clusterip-service-0e178a7a2aa150a6da237a6cde996688cd7c42f2120ff5adc3f2c858cde82a66.log
[2024/12/09 11:36:08] [debug] [input:tail:tail.0] excluded=/var/log/containers/slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_anypoint-monitoring-6f50734f5174b3710809d8fe3ae1a5e07dc8e249d948ec89413a7d21f0ad163f.log
[2024/12/09 11:36:08] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_app-63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d.log, inode 4387407
[2024/12/09 11:36:08] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_init-8b2f1341fc8719f7d592f93ecebdcb1b298940d8247fb07376806ce31a6da266.log, inode 4387388
[2024/12/09 11:36:08] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2024/12/09 11:36:15] [debug] [input:tail:tail.0] inode=4387407, /var/log/containers/slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_app-63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d.log, events: IN_MODIFY
[2024/12/09 11:36:15] [ info] [filter:multiline:multiline.0] created new multiline stream for tail.0_kube.var.log.containers.slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_app-63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d.log
[2024/12/09 11:36:15] [debug] [filter:multiline:multiline.0] Created new ML stream for tail.0_kube.var.log.containers.slb-rtf-aks-new-65bcf5f4f4-tvtjc_ORG_ID_app-63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d.log
[2024/12/09 11:36:15] [debug] [filter:kubernetes:kubernetes.1] Send out request to API Server for pods information
[2024/12/09 11:36:15] [debug] [http_client] not using http_proxy for header
[2024/12/09 11:36:15] [debug] [http_client] server kubernetes.default.svc:443 will close connection #88
[2024/12/09 11:36:15] [debug] [filter:kubernetes:kubernetes.1] Request (ns=ORG_ID, pod=slb-rtf-aks-new-65bcf5f4f4-tvtjc) http_do=0, HTTP Status: 200
[2024/12/09 11:36:15] [debug] [filter:kubernetes:kubernetes.1] could not merge JSON, root_type=3
[2024/12/09 11:36:15] [debug] [task] created task=0x7f030d6384e0 id=0 OK
[2024/12/09 11:36:15] [debug] [output:es:es.0] task_id=0 assigned to thread #0
{"create":{"_index":"*****-2024.12"}}
{"@timestamp":"2024-12-09T11:36:15.016Z","time":"2024-12-09T11:36:15.016955327Z","stream":"stdout","logtag":"F","log":"[2024-12-09 11:36:14.969] INFO  LoggerMessageProcessor [[MuleRuntime].uber.809: [slb-rtf-aks-new].slb-rtf-aksFlow.CPU_INTENSIVE @5bba7646] [event: cc4eb160-b621-11ef-bdc5-5afa9c807615]: {","kubernetes":{"pod_name":"slb-rtf-aks-new-65bcf5f4f4-tvtjc","namespace_name":"ORG_ID","pod_id":"37e3de46-c5ae-45bb-be22-316a08d60ef4","labels":{"am-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","app":"slb-rtf-aks-new","environment":"ORG_ID","name":"slb-rtf-aks-new","organization":"9357e0d4-ca00-43fe-85fe-f19e0c46badb","pod-template-hash":"65bcf5f4f4","root-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","rtf_mulesoft_com/disableAmLogForwarding":"true","rtf_mulesoft_com/generation":"3af3e3c0d051d9367963c3bba48f3c16","rtf_mulesoft_com/id":"bf57d697-21f6-4b08-9f51-cdb5b2d14abc","type":"MuleApplication"},"host":"aks-mulesoftpool-36227775-vmss000002","pod_ip":"10.244.2.129","container_name":"app","docker_id":"63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d","container_hash":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9@sha256:941784f7ee4188f69a58abd7671eda01c099dc09b4ce85d46c6bfc1ba4f7ecbd","container_image":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9:11-java17"}}
{"create":{"_index":"*****-2024.12"}}
{"@timestamp":"2024-12-09T11:36:15.016Z","time":"2024-12-09T11:36:15.016999127Z","stream":"stdout","logtag":"F","log":"  \"status\": \"alive\"","kubernetes":{"pod_name":"slb-rtf-aks-new-65bcf5f4f4-tvtjc","namespace_name":"ORG_ID","pod_id":"37e3de46-c5ae-45bb-be22-316a08d60ef4","labels":{"am-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","app":"slb-rtf-aks-new","environment":"ORG_ID","name":"slb-rtf-aks-new","organization":"9357e0d4-ca00-43fe-85fe-f19e0c46badb","pod-template-hash":"65bcf5f4f4","root-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","rtf_mulesoft_com/disableAmLogForwarding":"true","rtf_mulesoft_com/generation":"3af3e3c0d051d9367963c3bba48f3c16","rtf_mulesoft_com/id":"bf57d697-21f6-4b08-9f51-cdb5b2d14abc","type":"MuleApplication"},"host":"aks-mulesoftpool-36227775-vmss000002","pod_ip":"10.244.2.129","container_name":"app","docker_id":"63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d","container_hash":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9@sha256:941784f7ee4188f69a58abd7671eda01c099dc09b4ce85d46c6bfc1ba4f7ecbd","container_image":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9:11-java17"}}
{"create":{"_index":"*****-2024.12"}}
{"@timestamp":"2024-12-09T11:36:15.017Z","time":"2024-12-09T11:36:15.017003827Z","stream":"stdout","logtag":"F","log":"}","kubernetes":{"pod_name":"slb-rtf-aks-new-65bcf5f4f4-tvtjc","namespace_name":"ORG_ID","pod_id":"37e3de46-c5ae-45bb-be22-316a08d60ef4","labels":{"am-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","app":"slb-rtf-aks-new","environment":"ORG_ID","name":"slb-rtf-aks-new","organization":"9357e0d4-ca00-43fe-85fe-f19e0c46badb","pod-template-hash":"65bcf5f4f4","root-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","rtf_mulesoft_com/disableAmLogForwarding":"true","rtf_mulesoft_com/generation":"3af3e3c0d051d9367963c3bba48f3c16","rtf_mulesoft_com/id":"bf57d697-21f6-4b08-9f51-cdb5b2d14abc","type":"MuleApplication"},"host":"aks-mulesoftpool-36227775-vmss000002","pod_ip":"10.244.2.129","container_name":"app","docker_id":"63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d","container_hash":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9@sha256:941784f7ee4188f69a58abd7671eda01c099dc09b4ce85d46c6bfc1ba4f7ecbd","container_image":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9:11-java17"}}
{"create":{"_index":"*****-2024.12"}}
[2024/12/09 11:36:15] [debug] [upstream] KA connection #88 to xxx.xxx.xxx.xxx:xxxx is connected
[2024/12/09 11:36:15] [debug] [out_es] converted_size is 0
[2024/12/09 11:36:15] [debug] [http_client] not using http_proxy for header
{"@timestamp":"2024-12-09T11:36:15.017Z","time":"2024-12-09T11:36:15.017007227Z","stream":"stdout","logtag":"F","log":"ERROR StatusConsoleListener Attempted to append to non-started appender AnypointMonitoringFileAppender","kubernetes":{"pod_name":"slb-rtf-aks-new-65bcf5f4f4-tvtjc","namespace_name":"ORG_ID","pod_id":"37e3de46-c5ae-45bb-be22-316a08d60ef4","labels":{"am-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","app":"slb-rtf-aks-new","environment":"ORG_ID","name":"slb-rtf-aks-new","organization":"9357e0d4-ca00-43fe-85fe-f19e0c46badb","pod-template-hash":"65bcf5f4f4","root-org-id":"eb34a0c7-4457-421d-8fbc-543704629b56","rtf_mulesoft_com/disableAmLogForwarding":"true","rtf_mulesoft_com/generation":"3af3e3c0d051d9367963c3bba48f3c16","rtf_mulesoft_com/id":"bf57d697-21f6-4b08-9f51-cdb5b2d14abc","type":"MuleApplication"},"host":"aks-mulesoftpool-36227775-vmss000002","pod_ip":"10.244.2.129","container_name":"app","docker_id":"63550b94030d594da8ca0f2288359272aa9c061a2b3c2ae8ceff55afff0a007d","container_hash":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9@sha256:941784f7ee4188f69a58abd7671eda01c099dc09b4ce85d46c6bfc1ba4f7ecbd","container_image":"rtf-runtime-registry.kprod.msap.io/mulesoft/poseidon-runtime-4.6.9:11-java17"}}
[2024/12/09 11:36:15] [debug] [output:es:es.0] HTTP Status=200 URI=/_bulk
[2024/12/09 11:36:15] [debug] [output:es:es.0] Elasticsearch response
{"took":43,"ingest_took":0,"errors":false,"items":[{"create":{"_index":"*****-2024.12","_id":"lnI0q5MB8y8i7NOBqPqL","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":387191,"_primary_term":1,"status":201}},{"create":{"_index":"*****-2024.12","_id":"l3I0q5MB8y8i7NOBqPqL","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":387192,"_primary_term":1,"status":201}},{"create":{"_index":"*****-2024.12","_id":"mHI0q5MB8y8i7NOBqPqL","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":387193,"_primary_term":1,"status":201}},{"create":{"_index":"*****-2024.12","_id":"mXI0q5MB8y8i7NOBqPqL","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":387194,"_primary_term":1,"status":201}}]}
[2024/12/09 11:36:15] [debug] [upstream] KA connection #88 to xxx.xxx.xxx.xxx:xxxx is now available
[2024/12/09 11:36:15] [debug] [out flush] cb_destroy coro_id=0

我尝试过的

  1. 流畅的位输入配置：

    • 我使用 tail 输入插件从 pod 的日志文件中收集日志。

  2. 多行解析器：我尝试在 

    MULTILINE_PARSER
    中使用 
    custom_parsers.conf
    但仍然遇到问题。这是我当前的解析器：

     [PARSER]
     Name cri
     Format regex
     Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
     Time_Key    time
     Time_Format %Y-%m-%dT%H:%M:%S.%L%z
     Time_Keep   On

  [PARSER]
     Name        docker
     Format      json
     Time_Key    time
     Time_Format %Y-%m-%dT%H:%M:%S.%L
     Time_Keep   On
     Decode_Field    json     log


 [MULTILINE_PARSER]
     name          appParser
     type          regex
     key_content   log
     flush_timeout 1000
     # rules   |   state name   | regex pattern                 | next state name
     # --------|----------------|-----------------------------------------------------------------------
     rule         "start_state"   "/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{9}Z stdout [PF] \[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}\] (?:INFO|DEBUG|ERROR|WARNING|TRACE).*.\[\[MuleRuntime\].uber.*: \[.*.\](.*)/"  "cont"
     rule         "cont"          "/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{9}Z stdout [F] (?!.*INFO|TRACE|DEBUG|ERROR|WARNING)(.*)/"  "cont"

  3. 更新了 Fluent Bit 配置：这是我的 Fluent Bit 配置的相关部分：

     [SERVICE]
     Daemon Off
     Flush 1
     Log_Level debug
     Parsers_File custom_parsers.conf
     HTTP_Server On
     HTTP_Listen 0.0.0.0
     HTTP_Port 2020
     Health_Check On

 [INPUT]
     Name              tail
     Tag               kube.*
     Path              /var/log/containers/*.log
     multiline.parser  appServer
     DB                /var/log/flb_kube.db
     Mem_Buf_Limit     50MB
     Read_from_head    true
     Skip_Long_Lines   Off
     Refresh_Interval  10
     Rotate_Wait       10
     Exclude_Path      /var/log/containers/*_kube-system_*.log, /var/log/containers/*_monitoring_*.log, /var/log/containers/*_rtf_*.log, /var/log/containers/*_logging_*.log, /var/log/containers/*_calico-system_*.log, /var/log/containers/*_dynatrace_*.log, /var/log/containers/*_ingress-nginx_*.log, /var/log/containers/*_anypoint-monitoring*.log

 [FILTER]
     Name                kubernetes
     Match               kube.*
     Annotations         Off
     Merge_Log           On
     Keep_Log            On
     Labels              On
     Kube_Tag_Prefix     kube.var.log.containers.
     K8S-Logging.Parser  On
     K8S-Logging.Exclude On

 [OUTPUT]
     Name                es
     Match               *
     Host                *****
     Port                *****
     Index               *****
     tls                 On
     tls.verify          Off
     HTTP_User           *****
     HTTP_Passwd         *****
     Logstash_Format     On
     Logstash_Prefix     *****
     Logstash_DateFormat %Y.%m
     Retry_Limit         6
     Suppress_Type_Name  On
     Trace_Error         On
     Trace_Output        On
     Replace_Dots        On
     Include_Tag_Key     Off
     Current_Time_Index  Off

环境详情

  • Fluent Bit 版本：3.2.2
  • MuleSoft 运行时结构：在 AKS 中运行
  • Kubernetes 版本：1.30.6
  • Elasticsearch 版本：8.x

问题

  1. 如何配置 Fluent Bit 来正确处理这些多行日志并确保 JSON 负载不会被换行符分割？
  2. 是否有更好的方法或替代配置来处理具有 JSON 负载的多行日志？

其他信息 如果需要，我可以分享有关我的 Fluent Bit 设置或正在生成的日志的更多详细信息。

我尝试应用多行解析器但没有成功。

我还设法只应用 cri 解析器，但它也会拆分 json 换行符。

azure-aks mulesoft fluent-bit anypoint-rtf
1个回答
0
投票

经过一番尝试和错误，我找出了问题所在：

  1. 必须更新
    MULTILINE_PARSER
    中的规则，不考虑时间、流和日志标签
  2. 在
    MULTILINE_PARSER
    中添加解析器标志并指向cri，它将解析每一行并删除时间、流和日志标签

这样多行解析器就会知道要合并哪些行。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.