fluidd 输出到 s3 失败

问题描述 投票:0回答:1

我尝试使用 Fluentd 将 docker-compose 上的应用程序生成的日志放入我的 s3 存储桶,但出现以下错误

fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:456:in `rescue in check_apikeys'    
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:451:in `check_apikeys'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.3.0/lib/fluent/plugin/out_s3.rb:240:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
fluentd          | 2024-07-04 09:01:48 +0000 [error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::S3::Errors::AccessDenied: Access Denied>"
fluentd          |   2024-07-04 09:01:48 +0000 [error]: #0 suppressed same stacktrace
fluentd          | 2024-07-04 09:01:48 +0000 [error]: Worker 0 finished unexpectedly with status 1

为了编写输出到 s3 的配置文件,我遵循了 s3 插件的 github 指南

<match pattern>
  @type s3

  aws_key_id x
  aws_sec_key x
  s3_bucket x
  s3_region x

  path logs/${tag}/%Y/%m/%d/
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

  # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
  # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
  <buffer tag,time>
    @type file
    path /var/log/fluent/s3
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc true # use utc
  </buffer>
  <format>
    @type json
  </format>
</match>

我仔细检查了我的存储桶、可用区和密钥的拼写,一切都正确。 我错过了什么?

amazon-s3 docker-compose fluentd
1个回答
0
投票

已解决:除了简单的 PutObject 之外,Fluentd 还需要一组权限:它需要 PutObject、GetObject、ListBucket 和 ListBucketVersions

© www.soinside.com 2019 - 2024. All rights reserved.