aks 内部负载均衡器无法正常工作

Question

我尝试在启用内部负载均衡器的情况下部署 AKS。

内部负载均衡器由 istio 入口网关部署。

profile.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-controlplane
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
      k8s:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
        serviceAnnotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
        service:
          type: LoadBalancer
          loadBalancerIP: 10.1.2.254
          ports:
            - port: 5000
              targetPort: 5000
              name: store-front
            - port: 5001
              targetPort: 5001
              name: web-service
    pilot:
      k8s:
        env:
          - name: PILOT_TRACE_SAMPLING
            value: "100"
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
  values:
    global:
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
    pilot:
      autoscaleEnabled: false
    gateways:
      istio-egressgateway:
        autoscaleEnabled: false
      istio-ingressgateway:
        autoscaleEnabled: false

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: app-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 5000
      name: sample
      protocol: HTTP
    hosts:
      - "*"
  - port:
      number: 5001
      name: app
      protocol: HTTP
    hosts:
      - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: store-front
spec:
  hosts:
  - "*"
  gateways:
  - app-gateway
  http:
  - match:
    - uri:
        prefix: /sample
    route:
    - destination:
        host: store-front.default.svc.cluster.local
        port:
          number: 5000
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: web-service
spec:
  hosts:
  - "*"
  gateways:
  - app-gateway
  http:
  - match:
    - uri:
        prefix: /app
    route:
    - destination:
        host: web-service.default.svc.cluster.local
        port:
          number: 5001

当我尝试通过 VNet 对等从虚拟机连接到 http://10.1.2.254/app 或 http://10.1.2.254/sample 时，它不起作用。

所以我部署了一个测试pod并尝试curl，它也不起作用。

VirtualService 似乎无法正常工作。

这有什么原因吗？

kubectl version: v1.30.0

istioctl version: 1.22.3

Answer 1

VirtualService 可能无法正常工作。您可以检查 Istio ingress gateway pod 的日志，看看是否存在与 VirtualService 相关的错误。要使用 Istio 和内部负载均衡器设置 AKS 集群，请按照以下步骤操作 -

验证版本

您甚至可以在设置集群时从 aks 门户启用 istio

Istio pod 正在运行，下一步是使用内部负载均衡器配置 Istio Ingress Gateway

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: aks-istio-system
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    - port: 80
      targetPort: 80
      name: http2
    - port: 443
      targetPort: 443
      name: https
    - port: 5000
      targetPort: 5000
      name: store-front
    - port: 5001
      targetPort: 5001
      name: web-service

接下来我们将部署您的示例应用程序 store-front.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: store-front
  labels:
    app: store-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: store-front
  template:
    metadata:
      labels:
        app: store-front
    spec:
      containers:
      - name: store-front
        image: nginx
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: store-front
spec:
  ports:
  - port: 5000
  selector:
    app: store-front

我们还必须为 Web 服务创建部署 YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
  labels:
    app: web-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-service
  template:
    metadata:
      labels:
        app: web-service
    spec:
      containers:
      - name: web-service
        image: nginx
        ports:
        - containerPort: 5001
---
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  ports:
  - port: 5001
  selector:
    app: web-service

接下来我们需要配置 Istio 网关和虚拟服务

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: app-gateway
  namespace: aks-istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 5000
      name: sample
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 5001
      name: app
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: store-front
  namespace: aks-istio-system
spec:
  hosts:
  - "*"
  gateways:
  - app-gateway
  http:
  - match:
    - uri:
        prefix: /sample
    route:
    - destination:
        host: store-front.default.svc.cluster.local
        port:
          number: 5000
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: web-service
  namespace: aks-istio-system
spec:
  hosts:
  - "*"
  gateways:
  - app-gateway
  http:
  - match:
    - uri:
        prefix: /app
    route:
    - destination:
        host: web-service.default.svc.cluster.local
        port:
          number: 5001

如果 Ingress Gateway 未部署，请为其创建部署并确保其正确挂载根 CA 证书。如果您没有外部证书，您可以创建一个自签名证书并创建一个密钥

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-ingressgateway
  namespace: aks-istio-system
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-ingressgateway
      istio: ingressgateway
  template:
    metadata:
      labels:
        app: istio-ingressgateway
        istio: ingressgateway
    spec:
      containers:
      - name: istio-proxy
        image: docker.io/istio/proxyv2:1.22.3
        ports:
        - containerPort: 80
        - containerPort: 443
        - containerPort: 5000
        - containerPort: 5001
        args:
        - proxy
        - router
        - --domain
        - $(POD_NAMESPACE).svc.cluster.local
        - --log_output_level=default:info
        - --drainDuration
        - 45s
        - --parentShutdownDuration
        - 60s
        - --connectTimeout
        - 10s
        - --serviceCluster
        - istio-ingressgateway
        - --zipkinAddress
        - $(ZIPKIN)
        - --proxyLogLevel=warning
        - --proxyComponentLogLevel=misc:error
        - --dnsRefreshRate=300s
        - --statusPort
        - "15020"
        - --trust-domain=cluster.local
        - --controlPlaneAuthPolicy
        - MUTUAL_TLS
        - --discoveryAddress
        - istiod.aks-istio-system.svc:15012
        - --proxyAdminPort
        - "15000"
        - --concurrency
        - "2"
        - --controlPlaneBootstrap=true
        env:
        - name: JWT_POLICY
          value: third-party-jwt
        - name: PILOT_CERT_PROVIDER
          value: istiod
        - name: CA_ADDR
          value: istiod.aks-istio-system.svc:15012
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: CANONICAL_SERVICE
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['service.istio.io/canonical-name']
        - name: CANONICAL_REVISION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['service.istio.io/canonical-revision']
        - name: SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceAccountName
        volumeMounts:
        - name: istio-ingressgateway-certs
          mountPath: /etc/istio/ingressgateway-certs
          readOnly: true
        - name: istio-certs
          mountPath: /etc/certs
          readOnly: true
        - name: istio-root-ca
          mountPath: /var/run/secrets/istio
          readOnly: true
      volumes:
      - name: istio-ingressgateway-certs
        secret:
          secretName: istio-ingressgateway-certs
          optional: true
      - name: istio-certs
        secret:
          secretName: istio.istio-system
      - name: istio-root-ca
        secret:
          secretName: istio.istio-system

验证 Pod 状态

获取内部负载均衡器IP

测试服务（如果您是从虚拟机内部执行此操作，那么正如您所说，它必须位于同一个 vnet 中，即必须执行 vnet 对等互连）

我建议进行一些额外的检查-

验证 DNS 解析。

kubectl run -i --tty --rm dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 --restart=Never -- nslookup store-front.default.svc.cluster.local
kubectl run -i --tty --rm dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 --restart=Never -- nslookup web-service.default.svc.cluster.local

检查是否有任何网络策略阻止

kubectl get networkpolicy -A

参考资料：

• 用于 Istio 服务网格插件部署的 Azure Kubernetes 服务 (AKS) 外部或内部入口
• 排查 AKS 群集中的网络问题