Kiali 未与 VictoriaMetricsHello

问题描述 投票：0回答：2

有人能够使用 VictoriaMetrics 而不是 Prometheus 使 kiali 可视化网格吗？

当我使用 istio 示例中的 prometheus 和 kiali 设置时，Mesh Visualization 就可以工作了。

但是当我用 Victoria Metrics（代理、选择、插入、存储）替换 prometheus 时，kiali 只显示空图。

我检查过 Prometheus 和 Victoria Metrics 都具有相同的 istio_requests_total 指标。

但是当我在 spec.external_services.prometheus.url 中使用 Victoria Metrics 选择 URL 时，图表变为空。

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: anonymous
  external_services:
    istio:
      config_map_name: istio-1-14
      url_service_version: http://istiod-1-14:15014/version
    prometheus:
      url: http://vmselect-example-vmcluster-persistent.poc.svc.cluster.local:8481/select/0/prometheus/

在日志中，我看到两个与 vm-select 没有相应端点的事实相关的错误

2022-07-15T19:25:13Z ERR Failed to fetch Prometheus configuration:     bad_response: readObjectStart: expect { or n, but found r, error found      in #1 byte of ...|remoteAddr:|..., bigger context ...|remoteAddr:      "10.4.34.83:57468"; requestURI: /select|... 
2022-07-15T19:25:13Z ERR Failed to fetch Prometheus flags: bad_response: readObjectStart:     expect { or n, but found r, error found in #1 byte of     ...|remoteAddr:|..., bigger context ...|remoteAddr:     "10.4.34.83:57468"; requestURI: /select|...

和多个警告

 2022-07-15T19:35:28Z WRN Skipping      {destination_canonical_revision="v1",     destination_canonical_service="microservice",      destination_cluster="Kubernetes",     destination_service="microservice.poc.svc.cluster.local",      destination_service_name="microservice",     destination_service_namespace="poc",     destination_workload="microservice",     destination_workload_namespace="poc", request_protocol="http",     response_code="200", response_flags="-", source_canonical_revision="latest",     source_canonical_service="istio-ingressgateway-internal",  source_cluster="Kubernetes"}, missing expected TS labels

这是我的 VMPodScrape，我希望它能获取所有 pod 的所有指标

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: all-scrape
  namespace: poc
spec:
  podMetricsEndpoints:
    - scheme: http
      path: /stats/prometheus
      targetPort: 15090
  selector: {}
  namespaceSelector:
    any: true
istio kiali victoriametrics
2个回答
2
投票

错误消息看起来并不重要，如果出现错误，kiali 应该使用默认值。据我了解，它试图根据普罗米修斯配置文件和标志来识别抓取间隔和保留时间。

我认为您在重新标记配置时遇到问题，它会删除 kiali 所需的标签。

有按指标名称需要标签的文档： https://kiali.io/docs/faq/general/#which-istio-metrics-and-attributes-are-required-by-kiali

我建议检查 VMAgent 的抓取配置。可能重新标记规则已经过时了。

0
投票

对我来说，我没有使用 VMPodScrape，

我根据官方指南（https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single.html）通过Helm图表在Kubernetes中使用了最简单的VictoriaMetrics Single 注意：我在“victoria-metric”命名空间上安装。

安装后，我编辑configmap，它基于 istio-1.17.1/samples/addons/prometheus.yaml ，配置图如下

kind: ConfigMap
apiVersion: v1
metadata:
  annotations:
    meta.helm.sh/release-name: vmsingle
    meta.helm.sh/release-namespace: victoria-metric
  labels:
    app: server
    app.kubernetes.io/instance: vmsingle
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-single
    helm.sh/chart: victoria-metrics-single-0.8.55
  name: vmsingle-victoria-metrics-single-server-scrapeconfig
  namespace: victoria-metric
data:
  scrape.yml: |2

    global:
      scrape_interval: 15s

    scrape_configs:
    - job_name: victoriametrics
      static_configs:
      - targets:
        - localhost:8428
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-apiservers
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: default;kubernetes;https
        source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
        - __meta_kubernetes_endpoint_port_name
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - honor_labels: true
      job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: drop
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: service
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node
    - honor_labels: true
      job_name: kubernetes-service-endpoints-slow
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: service
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node
      scrape_interval: 5m
      scrape_timeout: 30s
    - honor_labels: true
      job_name: prometheus-pushgateway
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - action: keep
        regex: pushgateway
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
    - honor_labels: true
      job_name: kubernetes-services
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module:
        - http_2xx
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
      - source_labels:
        - __address__
        target_label: __param_target
      - replacement: blackbox
        target_label: __address__
      - source_labels:
        - __param_target
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - source_labels:
        - __meta_kubernetes_service_name
        target_label: service
    - honor_labels: true
      job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: drop
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: drop
        regex: Pending|Succeeded|Failed|Completed
        source_labels:
        - __meta_kubernetes_pod_phase
    - honor_labels: true
      job_name: kubernetes-pods-slow
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: drop
        regex: Pending|Succeeded|Failed|Completed
        source_labels:
        - __meta_kubernetes_pod_phase
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

这是我的 kiali 配置图。

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    version: v1.63.1
  name: kiali
  namespace: istio-system
data:
  config.yaml: |
    ...
    external_services:
      custom_dashboards:
        enabled: true
      istio:
        root_namespace: istio-system
      prometheus:
        url: http://vmsingle-victoria-metrics-single-server.victoria-metric.svc.cluster.local:8428/

我的设置中一切正常。

在我调试它的过程中，您可以将流量转发到 victoria_metric pod 以查看调试 WebUI

kubectl port-forward <victoria_metric_pod_name> 8428:8428

然后访问 http://localhost:8428/targets 上的 VictoriaMetric 该页面将在此处列出 scape 目标，如果您正确配置了 victoria metric scape，我们应该会看到“kubernetes-pods”作业启动并运行。

这是我设置中的 Kiali 图。 请注意，在 Kiali pod 中，日志中有一些错误……但图表仍在显示。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.