尽管在 Istio 1.21.4 中进行了遥测配置，但仍无法删除 Prometheus 中的特定标签和指标

Question

---
apiVersion: install.istio.io/v1alpha1 
kind: IstioOperator
metadata:
  <some metadata>
spec:
  <some specifications>
  values:
    global:
      # Ensure that the Istio pods are only scheduled to run on Linux nodes
      autoscalingv2API: true
      defaultNodeSelector:
        beta.kubernetes.io/os: linux
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi     
        autoInject: "disabled"
      imagePullPolicy: "IfNotPresent"
      imagePullSecrets:
      <some secrets>
    sidecarInjectorWebhook:
    <some values>
    telemetry:
      v2:
        prometheus:
          configOverride:
            inboundSidecar:
              debug: false
              stat_prefix: istio
              metrics:
                # Remove these tags from all Istio metrics
                - tags_to_remove: ["array of tags"]
                # Remove response tags from some metrics
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]
            outboundSidecar:
              debug: false
              stat_prefix: istio
              metrics:
                # Remove these tags from all Istio metrics
                - tags_to_remove: ["array of tags"]
                # Remove response tags from some metrics
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]
            gateway:
              debug: false
              stat_prefix: istio
              disable_host_header_fallback: true
              metrics:
                # Remove these tags from all Istio metrics
                - tags_to_remove: ["array of tags"]
                # Remove response tags from some metrics
                - name: request_duration_milliseconds
                  tags_to_remove: ["response_code", "response_flags"]
                - name: request_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                - name: response_bytes
                  tags_to_remove: ["response_code", "response_flags"]
                  

这之前是有效的，但升级到版本 1.21.4 后，我收到一条错误，指出 configOverride 无效。没有好的文档告诉我如何解决这个问题。

我已经使用了这样的遥测，但仍然没有从普罗米修斯中删除标签。我有什么遗漏吗？

---
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-telemetry-outbound
  namespace: istio-system
spec:
  metrics:
    - providers:
        - name: prometheus
          overrides:
            # Apply to all outbound sidecars
            - match:
                mode: CLIENT_AND_SERVER
              tagOverrides:
                destination_canonical_service: 
                  operation: REMOVE
                source_canonical_service: 
                  operation: REMOVE
                destination_principal: 
                  operation: REMOVE
                source_principal: 
                  operation: REMOVE
                connection_security_policy:
                  operation: REMOVE
                grpc_response_status: 
                  operation: REMOVE
                source_version: 
                  operation: REMOVE
                destination_version:
                  operation: REMOVE
                request_protocol: 
                  operation: REMOVE
                source_canonical_revision: 
                  operation: REMOVE
                destination_canonical_revision: 
                  operation: REMOVE

            # Remove response tags from specific metrics
            - match:
                metric: REQUEST_DURATION
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

            - match:
                metric: REQUEST_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

            - match:
                metric: RESPONSE_SIZE
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code: 
                  operation: REMOVE
                response_flags: 
                  operation: REMOVE

Answer 1

检查命名空间 istio-system 中是否还有其他遥测资源。似乎创建新的遥测数据不会与同一命名空间中预先存在的遥测数据合并。

例如，

我有这个网格默认遥测。

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: otel
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 100

然后我为指标创建另一个，由于已有的指标而没有任何影响。

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: remove-dim
  namespace: istio-system
spec:
  metrics:
  - overrides:
    - match:
        metric: REQUEST_COUNT
        mode: CLIENT_AND_SERVER
      tagOverrides:
        destination_canonical_service:
          operation: REMOVE
    providers:
    - name: prometheus

为了使新的遥测生效， 编辑网格-默认遥测

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: otel
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 100
  metrics:
  - overrides:
    - match:
        metric: REQUEST_COUNT
        mode: CLIENT_AND_SERVER
      tagOverrides:
        destination_canonical_service:
          operation: REMOVE

或者在目标命名空间中创建一个。

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: remove-dimension
  namespace: test
spec:
  metrics:
  - overrides:
    - match:
        metric: REQUEST_COUNT
        mode: CLIENT_AND_SERVER
      tagOverrides:
        destination_canonical_service:
          operation: REMOVE