无法附加或装载卷：等待条件超时 - AKS Solr 服务器错误

Question

我尝试在 AKS 集群上运行 solr 服务器 Pod，但它不断抛出错误 -

Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[backup-repository-solr-backup kube-api-access-mhz4q solr-xml scgnfs-solr-pvc]: timed out waiting for the condition

。我已经使用 Terraform 部署了集群、卷挂载和其他资源。

一些 kubectl 输出如图所示 -

$ kubectl get pods -n <namespace>

NAME                                               READY   STATUS     RESTARTS        AGE
apache-solr-server-solrcloud-0                     0/1     Init:0/1   0               3d14h
apache-solr-server-solrcloud-1                     0/1     Init:0/1   0               3d14h
apache-solr-server-solrcloud-2                     0/1     Init:0/1   0               3d14h
apache-solr-server-solrcloud-zookeeper-0           1/1     Running    0               3d14h
apache-solr-server-solrcloud-zookeeper-1           1/1     Running    1 (3d14h ago)   3d14h
apache-solr-server-solrcloud-zookeeper-2           1/1     Running    4 (25h ago)     3d14h
solr-operator-7c6d46db-qsp4t                       1/1     Running    0               3d14h
solr-operator-zookeeper-operator-cfc5d5d59-4hggr   1/1     Running    0               3d14h

$ kubectl describe pod <pod_name> -n <namespace>

Name:             apache-solr-server-solrcloud-0
Namespace:        cs-solr
Priority:         0
Service Account:  default
Node:             aks-appsv1-14790217-vmss000010/172.21.145.100
Start Time:       Fri, 10 Nov 2023 21:16:32 +0530
Labels:           app.kubernetes.io/instance=apache-solr-server
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=solr
                  app.kubernetes.io/version=8.11.1
                  controller-revision-hash=apache-solr-server-solrcloud-c9cd4b88c
                  helm.sh/chart=solr-0.8.0
                  solr-cloud=apache-solr-server
                  statefulset.kubernetes.io/pod-name=apache-solr-server-solrcloud-0
                  technology=solr-cloud
Annotations:      solr.apache.org/backupRepositories: solr-backup
                  solr.apache.org/solrXmlMd5: 28062a041f9071bd95bcf3b0c9ce01b7
Status:           Pending
IP:
IPs:              <none>
Controlled By:    StatefulSet/apache-solr-server-solrcloud
Init Containers:
  cp-solr-xml:
    Container ID:
    Image:         library/busybox:1.28.0-glibc
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      cp /tmp/solr.xml /tmp-config/solr.xml && addgroup -g 8983 solr && adduser -u 8983 -G solr -H -D solr && (su solr -c 'test -w /var/solr/data/backup-restore/solr-backup' || chown -R 8983:8983 /var/solr/data/backup-restore/solr-backup)
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     50e-3
      memory:  50M
    Requests:
      cpu:        50e-3
      memory:     50M
    Environment:  <none>
    Mounts:
      /tmp from solr-xml (rw)
      /tmp-config from scgnfs-solr-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhz4q (ro)
      /var/solr/data/backup-restore/solr-backup from backup-repository-solr-backup (rw,path="cloud/apache-solr-server")
Containers:
  solrcloud-node:
    Container ID:
    Image:          solr:8.11
    Image ID:
    Port:           8983/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8983/solr/admin/info/system delay=0s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8983/solr/admin/info/health delay=0s timeout=1s period=10s #success=1 #failure=2
    Startup:        http-get http://:8983/solr/admin/info/system delay=10s timeout=1s period=5s #success=1 #failure=10
    Environment:
      SOLR_JAVA_MEM:        -Xms1g -Xmx2g
      SOLR_HOME:            /var/solr/data
      SOLR_PORT:            8983
      SOLR_NODE_PORT:       8983
      SOLR_PORT_ADVERTISE:  8983
      POD_HOSTNAME:         apache-solr-server-solrcloud-0 (v1:metadata.name)
      POD_NAME:             apache-solr-server-solrcloud-0 (v1:metadata.name)
      POD_IP:                (v1:status.podIP)
      POD_NAMESPACE:        cs-solr (v1:metadata.namespace)
      SOLR_HOST:            $(POD_NAME).apache-solr-server-solrcloud-headless.cs-solr
      SOLR_LOG_LEVEL:       INFO
      GC_TUNE:
      SOLR_STOP_WAIT:       55
      ZK_HOST:              apache-solr-server-solrcloud-zookeeper-0.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-1.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-2.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181/
      ZK_CHROOT:            /
      ZK_SERVER:            apache-solr-server-solrcloud-zookeeper-0.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-1.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-2.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181
      SOLR_OPTS:            -DhostPort=$(SOLR_NODE_PORT)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhz4q (ro)
      /var/solr/data from scgnfs-solr-pvc (rw)
      /var/solr/data/backup-restore/solr-backup from backup-repository-solr-backup (rw,path="cloud/apache-solr-server")
Readiness Gates:
  Type                           Status
  solr.apache.org/isNotStopped   True
Conditions:
  Type                           Status
  solr.apache.org/isNotStopped   True
  Initialized                    False
  Ready                          False
  ContainersReady                False
  PodScheduled                   True
Volumes:
  scgnfs-solr-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  scgnfs-solr-pvc-apache-solr-server-solrcloud-0
    ReadOnly:   false
  solr-xml:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      apache-solr-server-solrcloud-configmap
    Optional:  false
  backup-repository-solr-backup:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  scgnfs-solr-pvc
    ReadOnly:   false
  kube-api-access-mhz4q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                      From     Message
  ----     ------       ----                     ----     -------
  Warning  FailedMount  19m (x979 over 3d13h)    kubelet  Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[solr-xml scgnfs-solr-pvc backup-repository-solr-backup kube-api-access-mhz4q]: timed out waiting for the condition
  Warning  FailedMount  10m (x437 over 3d14h)    kubelet  Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[scgnfs-solr-pvc backup-repository-solr-backup kube-api-access-mhz4q solr-xml]: timed out waiting for the condition
  Warning  FailedMount  3m29s (x431 over 3d13h)  kubelet  Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[backup-repository-solr-backup kube-api-access-mhz4q solr-xml scgnfs-solr-pvc]: timed out waiting for the condition

$ kubectl get pvc -n <namespace>

NAME                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            AGE
data-apache-solr-server-solrcloud-zookeeper-0    Bound    pvc-a520af21-4084-4af5-b954-4613697a43ac   20Gi       RWO            default                 3d14h
data-apache-solr-server-solrcloud-zookeeper-1    Bound    pvc-3f27a00f-6577-4126-a014-f13c5ab4b881   20Gi       RWO            default                 3d14h
data-apache-solr-server-solrcloud-zookeeper-2    Bound    pvc-5e9a1e38-7f74-4270-9ac6-02df453190e4   20Gi       RWO            default                 3d14h
scgnfs-solr-pvc                                  Bound    scgnfs-solr-pv                             100        RWX            azurefile-csi-premium   3d15h
scgnfs-solr-pvc-apache-solr-server-solrcloud-0   Bound    pvc-ba419a5f-f634-4aa5-97f4-01a6066bb227   100Gi      RWO            azurefile-csi-premium   3d14h
scgnfs-solr-pvc-apache-solr-server-solrcloud-1   Bound    pvc-2acf3702-3e38-4c89-9366-7cd8336aee88   100Gi      RWO            azurefile-csi-premium   3d14h
scgnfs-solr-pvc-apache-solr-server-solrcloud-2   Bound    pvc-c13b240b-e133-46a5-b2ba-50f12d0aea07   100Gi      RWO            azurefile-csi-premium   3d14h

$ kubectl describe pvc scgnfs-solr-pvc -n <namespace>

Name:          scgnfs-solr-pvc
Namespace:     cs-solr
StorageClass:  azurefile-csi-premium
Status:        Bound
Volume:        scgnfs-solr-pv
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       apache-solr-server-solrcloud-0
               apache-solr-server-solrcloud-1
               apache-solr-server-solrcloud-2
Events:        <none>

配置solr-server helm图表时传递的values.yaml文件-

version: 0.8.0
backupRepositories:
    - name: "solr-backup"
      volume:
        source:
          persistentVolumeClaim:
            claimName: "scgnfs-solr-pvc"
dataStorage:
   type: persistent 
   capacity: 100Gi 
   persistent:
     pvc:
       name: scgnfs-solr-pvc
       storageClassName: azurefile-csi-premium
resources:
  limits:
    cpu: 4000m
    memory: 4G

备份存储库的 PVC 与 solr-server 的 PVC 相同。这是一个问题吗？我不知道问题出在哪里。 PV和PVC似乎都已成功应用。

任何人都可以帮助我到底错在哪里吗？预先感谢。

Edit1：我为 backupRepository 创建了不同的 PV/PVC，但这也给出了相同的错误。

Answer 1

事件日志表明 kubelet 在为 Solr pod 连接或安装卷时遇到问题，并且 kubelet 超时。 请验证存储类是否已正确安装和配置。

kubectl get pods -n kube-system | grep azurefile

和

kubectl get storageclasses

这里是 azurefile-csi-premium。 我相信供应商有问题。

我正在使用类似的 helm 配置在 aks 上设置 Solr 并且它正在工作。

kubectl get pods

建议您验证与 pvc 关联的 pv 是否已正确配置，并且 azure 文件存储可以访问它，并验证 Azure 文件 CSI 驱动程序是否在集群中正确配置（如上所示）。

您可以参考

上的类似问题

kubernetes QnA 论坛

MS 文档

GitHub