Kibana/Logstash 不使用 grok 模式提取数据

Question

出于某种我不知道的原因，elasticsearch 不会使用 grok 模式从日志文件中提取数据：

input {
  file {
    path => "/mnt/tutorialdata/www1/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { 
        "message" => "%{IPORHOST:client} %{USER:ident} %{USER:auth} \[(?<timestamp>%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME})\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP %{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:bytes} \"%{URI:referrer}\" \"%{GREEDYDATA:agent}\" %{NUMBER:duration}"
    }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss" ]
  }
}


output {
  elasticsearch {
    user => "logstash_internal"
    password => "${LOGSTASH_INTERNAL_PASSWORD}"
    hosts => "elasticsearch:9200"
  }
}

样本数据：

209.160.24.63 - - [18/Mar/2024:18:22:16] "GET /product.screen?productId=WC-SH-A02&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 3878 "http://www.google.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 349
209.160.24.63 - - [18/Mar/2024:18:22:16] "GET /oldlink?itemId=EST-6&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 1748 "http://www.buttercupgames.com/oldlink?itemId=EST-6" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 731
209.160.24.63 - - [18/Mar/2024:18:22:17] "GET /product.screen?productId=BS-AG-G09&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 2550 "http://www.buttercupgames.com/product.screen?productId=BS-AG-G09" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 422
209.160.24.63 - - [18/Mar/2024:18:22:19] "POST /category.screen?categoryId=STRATEGY&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 407 "http://www.buttercupgames.com/cart.do?action=remove&itemId=EST-7&productId=PZ-SG-G05" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 211
209.160.24.63 - - [18/Mar/2024:18:22:20] "GET /product.screen?productId=FS-SG-G03&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 2047 "http://www.buttercupgames.com/category.screen?categoryId=STRATEGY" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 487
209.160.24.63 - - [18/Mar/2024:18:22:20] "POST /cart.do?action=addtocart&itemId=EST-21&productId=FS-SG-G03&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 1201 "http://www.buttercupgames.com/product.screen?productId=FS-SG-G03" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 256
209.160.24.63 - - [18/Mar/2024:18:22:21] "POST /cart.do?action=purchase&itemId=EST-21&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 486 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-21&categoryId=STRATEGY&productId=FS-SG-G03" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 293
209.160.24.63 - - [18/Mar/2024:18:22:22] "POST /cart/success.do?JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 3280 "http://www.buttercupgames.com/cart.do?action=purchase&itemId=EST-21" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 952
209.160.24.63 - - [18/Mar/2024:18:22:21] "GET /cart.do?action=remove&itemId=EST-11&productId=WC-SH-A01&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 3619 "http://www.buttercupgames.com/oldlink?itemId=EST-11" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 763
209.160.24.63 - - [18/Mar/2024:18:22:22] "GET /oldlink?itemId=EST-14&JSESSIONID=SD0SL6FF7ADFF4953 HTTP 1.1" 200 1352 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-14&productId=WC-SH-A01" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5" 180
112.111.162.4 - - [18/Mar/2024:18:26:36] "GET /product.screen?productId=WC-SH-G04&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 200 778 "http://www.buttercupgames.com/category.screen?categoryId=SHOOTER" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 194
112.111.162.4 - - [18/Mar/2024:18:26:37] "POST /cart.do?action=addtocart&itemId=EST-18&productId=WC-SH-G04&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 200 215 "http://www.buttercupgames.com/product.screen?productId=WC-SH-G04" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 727
112.111.162.4 - - [18/Mar/2024:18:26:38] "POST /cart.do?action=purchase&itemId=EST-18&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 200 1228 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-18&categoryId=SHOOTER&productId=WC-SH-G04" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 430
112.111.162.4 - - [18/Mar/2024:18:26:38] "POST /cart/error.do?msg=CreditDoesNotMatch&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 200 1232 "http://www.buttercupgames.com/cart.do?action=purchase&itemId=EST-18" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 841
112.111.162.4 - - [18/Mar/2024:18:26:37] "GET /category.screen?categoryId=NULL&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 505 2445 "http://www.buttercupgames.com/category.screen?categoryId=NULL" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 393
112.111.162.4 - - [18/Mar/2024:18:26:38] "GET /oldlink?itemId=EST-7&JSESSIONID=SD7SL8FF5ADFF4964 HTTP 1.1" 503 1207 "http://www.buttercupgames.com/category.screen?categoryId=NULL" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5" 704
74.125.19.106 - - [18/Mar/2024:18:32:15] "GET /cart.do?action=addtocart&itemId=EST-16&productId=DC-SG-G02&JSESSIONID=SD4SL7FF10ADFF4998 HTTP 1.1" 200 1425 "http://www.buttercupgames.com" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" 375
74.125.19.106 - - [18/Mar/2024:18:32:15] "GET /category.screen?categoryId=NULL&JSESSIONID=SD4SL7FF10ADFF4998 HTTP 1.1" 503 2039 "http://www.buttercupgames.com/oldlink?itemId=EST-13" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" 533
117.21.246.164 - - [18/Mar/2024:18:36:02] "POST /cart.do?action=changequantity&itemId=EST-21&productId=WC-SH-A01&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 809 "http://www.buttercupgames.com" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 643
117.21.246.164 - - [18/Mar/2024:18:36:03] "POST /cart.do?action=addtocart&itemId=EST-27&productId=DC-SG-G02&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 1291 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-27&productId=DC-SG-G02" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 795
117.21.246.164 - - [18/Mar/2024:18:36:03] "GET /category.screen?categoryId=STRATEGY&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 3182 "http://www.buttercupgames.com/oldlink?itemId=EST-26" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 190
117.21.246.164 - - [18/Mar/2024:18:36:03] "GET /cart.do?action=view&itemId=EST-19&productId=DB-SG-G01&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 2477 "http://www.buttercupgames.com/product.screen?productId=DB-SG-G01" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 636
117.21.246.164 - - [18/Mar/2024:18:36:05] "POST /product.screen?productId=DB-SG-G01&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 3792 "http://www.buttercupgames.com/cart.do?action=view&itemId=EST-7&productId=DB-SG-G01" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 360
117.21.246.164 - - [18/Mar/2024:18:36:06] "GET /category.screen?categoryId=ACCESSORIES&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 689 "http://www.buttercupgames.com/oldlink?itemId=EST-7" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 673
117.21.246.164 - - [18/Mar/2024:18:36:07] "GET /oldlink?itemId=EST-17&JSESSIONID=SD9SL6FF8ADFF5015 HTTP 1.1" 200 924 "http://www.buttercupgames.com/oldlink?itemId=EST-17" "Googlebot/2.1 (http://www.googlebot.com/bot.html)" 156

请注意，我已经在 Grok 调试器上成功测试了该模式，我可以看到包含所有匹配项的输出 JSON。但是，当我导航到 Kibana 上的 Observability/Logs/Stream 时，grok 列不存在（尽管我可以在左侧搜索它们，但没有具有这些值的行）。

Answer 1

好消息是问题不是来自你的 grok。

我在本地尝试过它，它的工作原理就像一个魅力（这是logstash stdout输出）：

{
       "referrer" => "http://www.buttercupgames.com",
       "response" => "200",
           "path" => "C:/path/to/my/access.log",
     "@timestamp" => 2024-03-18T17:32:15.000Z,
           "host" => "HOSTNAME",
        "message" => "74.125.19.106 - - [18/Mar/2024:18:32:15] \"GET /cart.do?action=addtocart&itemId=EST-16&productId=DC-SG-G02&JSESSIONID=SD4SL7FF10ADFF4998 HTTP 1.1\" 200 1425 \"http://www.buttercupgames.com\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6\" 375\r",
         "client" => "74.125.19.106",
          "ident" => "-",
          "agent" => "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6",
       "duration" => "375",
        "request" => "/cart.do?action=addtocart&itemId=EST-16&productId=DC-SG-G02&JSESSIONID=SD4SL7FF10ADFF4998",
         "method" => "GET",
    "httpversion" => "1.1",
       "@version" => "1",
          "bytes" => "1425",
           "auth" => "-",
      "timestamp" => "18/Mar/2024:18:32:15"
}

现在，如果您对同一个文件运行两次，第二次将被忽略，因为logstash将认为它已经计算出来。 为了说服自己，您可以尝试将日志文件重命名为

access2.log

，它应该可以正常工作。

现在文件输入插件对我来说仍然有点棘手，我无法进一步帮助您了解如何更改此行为。