使用DaemonSet运行glusterfs集群

Question

我一直在尝试使用以下方法在我的kubernetes集群上运行glusterfs集群：

glusterfs-service.json

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "glusterfs-cluster"
  },
  "spec": {
    "type": "NodePort",
    "selector": {
      "name": "gluster"
    },
    "ports": [
      {
        "port": 1
      }
    ]
  }
}

和glusterfs-server.json：

{
  "apiVersion": "extensions/v1beta1",
  "kind": "DaemonSet",
  "metadata": {
    "labels": {
      "name": "gluster"
    },
    "name": "gluster"
  },
  "spec": {
    "selector": {
      "matchLabels": {
        "name": "gluster"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "name": "gluster"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "gluster",
            "image": "gluster/gluster-centos",
            "livenessProbe": {
              "exec": {
                "command": [
                  "/bin/bash",
                  "-c",
                  "systemctl status glusterd.service"
                ]
              }
            },
            "readinessProbe": {
              "exec": {
                "command": [
                  "/bin/bash",
                  "-c",
                  "systemctl status glusterd.service"
                ]
              }
            },
            "securityContext": {
              "privileged": true
            },
            "volumeMounts": [
              {
                "mountPath": "/mnt/brick1",
                "name": "gluster-brick"
              },
              {
                "mountPath": "/etc/gluster",
                "name": "gluster-etc"
              },
              {
                "mountPath": "/var/log/gluster",
                "name": "gluster-logs"
              },
              {
                "mountPath": "/var/lib/glusterd",
                "name": "gluster-config"
              },
              {
                "mountPath": "/dev",
                "name": "gluster-dev"
              },
              {
                "mountPath": "/sys/fs/cgroup",
                "name": "gluster-cgroup"
              }
            ]
          }
        ],
        "dnsPolicy": "ClusterFirst",
        "hostNetwork": true,
        "volumes": [
          {
            "hostPath": {
              "path": "/mnt/brick1"
            },
            "name": "gluster-brick"
          },
          {
            "hostPath": {
              "path": "/etc/gluster"
            },
            "name": "gluster-etc"
          },
          {
            "hostPath": {
              "path": "/var/log/gluster"
            },
            "name": "gluster-logs"
          },
          {
            "hostPath": {
              "path": "/var/lib/glusterd"
            },
            "name": "gluster-config"
          },
          {
            "hostPath": {
              "path": "/dev"
            },
            "name": "gluster-dev"
          },
          {
            "hostPath": {
              "path": "/sys/fs/cgroup"
            },
            "name": "gluster-cgroup"
          }
        ]
      }
    }
  }
}

然后在我的pod定义中，我正在做：

"volumes": [
  {
    "name": "< volume name >",
    "glusterfs": {
      "endpoints": "glusterfs-cluster.default.svc.cluster.local",
      "path": "< gluster path >",
      "readOnly": false
    }
  }
]

但是pod创建已超时，因为它无法装入卷

它看起来只有一个glusterfs pod正在运行

这是我的日志：http://imgur.com/a/j2I8r

然后我尝试在我的gluster集群运行时在同一名称空间上运行我的pod，我现在收到此错误：

Operation for "\"kubernetes.io/glusterfs/01a0834e-64ab-11e6-af52-42010a840072-ssl-certificates\" (\"01a0834e-64ab-11e6-af52-42010a840072\")" failed.
No retries permitted until 2016-08-17 18:51:20.61133778 +0000 UTC (durationBeforeRetry 2m0s).
Error: MountVolume.SetUp failed for volume "kubernetes.io/glusterfs/01a0834e-64ab-11e6-af52-42010a840072-ssl-certificates" (spec.Name: "ssl-certificates") pod "01a0834e-64ab-11e6-af52-42010a840072" (UID: "01a0834e-64ab-11e6-af52-42010a840072") with: glusterfs: mount failed:
mount failed: exit status 1
Mounting arguments:
10.132.0.7:ssl_certificates /var/lib/kubelet/pods/01a0834e-64ab-11e6-af52-42010a840072/volumes/kubernetes.io~glusterfs/ssl-certificates
glusterfs [log-level=ERROR log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/ssl-certificates/caddy-server-1648321103-epvdi-glusterfs.log]
Output: Mount failed. Please check the log file for more details. the following error information was pulled from the glusterfs log to help diagnose this issue:
[2016-08-17 18:49:20.583585] E [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:ssl_certificates)
[2016-08-17 18:49:20.610531] E [glusterfsd-mgmt.c:1494:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server

Answer 1

日志清楚地说明发生了什么：

未能获得端点glusterfs-cluster [端点“glusterfs-cluster”未找到]

因为：

  "ports": [
  {
    "port": 1
  }

在某些方面是虚假的。首先，端口“1”非常可疑。第二，它在containerPort:方面没有匹配的DaemonSet，kubernetes可以指向Service - 因此，它不会为Endpoints元组创建(podIP, protocol, port)。因为glusterfs（合理地）想要直接接触底层Pods，而不通过Service，然后它无法发现Pods并且一切都突然停止。