无法使用etcd在vitess上打开topo服务器

Question

我正在用Helm运行一个简单的例子。请看下面的values.yaml文件：

cat << EOF | helm install helm/vitess -n vitess -f -
topology:
  cells:
    - name: 'zone1'
      keyspaces:
        - name: 'vitess'
          shards:
            - name: '0'
              tablets:
                - type: 'replica'
                  vttablet:
                    replicas: 1
      mysqlProtocol:
        enabled: true
        authType: secret
        username: vitess
        passwordSecret: vitess-db-password
      etcd:
        replicas: 3
      vtctld:
        replicas: 1
      vtgate:
        replicas: 3

vttablet:
  dataVolumeClaimSpec:
    storageClassName: nfs-slow
EOF

看看下面运行的当前pod的输出：

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS                  RESTARTS   AGE
kube-system   coredns-fb8b8dccf-8f5kt                  1/1     Running                 0          32m
kube-system   coredns-fb8b8dccf-qbd6c                  1/1     Running                 0          32m
kube-system   etcd-master1                             1/1     Running                 0          32m
kube-system   kube-apiserver-master1                   1/1     Running                 0          31m
kube-system   kube-controller-manager-master1          1/1     Running                 0          32m
kube-system   kube-flannel-ds-amd64-bkg9z              1/1     Running                 0          32m
kube-system   kube-flannel-ds-amd64-q8vh4              1/1     Running                 0          32m
kube-system   kube-flannel-ds-amd64-vqmnz              1/1     Running                 0          32m
kube-system   kube-proxy-bd8mf                         1/1     Running                 0          32m
kube-system   kube-proxy-nlc2b                         1/1     Running                 0          32m
kube-system   kube-proxy-x7cd5                         1/1     Running                 0          32m
kube-system   kube-scheduler-master1                   1/1     Running                 0          32m
kube-system   tiller-deploy-8458f6c667-cx2mv           1/1     Running                 0          27m
vitess        etcd-global-6pwvnv29th                   0/1     Init:0/1                0          16m
vitess        etcd-operator-84db9bc774-j4wml           1/1     Running                 0          30m
vitess        etcd-zone1-zwgvd7spzc                    0/1     Init:0/1                0          16m
vitess        vtctld-86cd78b6f5-zgfqg                  0/1     CrashLoopBackOff        7          16m
vitess        vtgate-zone1-58744956c4-x8ms2            0/1     CrashLoopBackOff        7          16m
vitess        zone1-vitess-0-init-shard-master-mbbph   1/1     Running                 0          16m
vitess        zone1-vitess-0-replica-0                 0/6     Init:CrashLoopBackOff   7          16m

运行日志我看到这个错误：

$ kubectl logs -n vitess vtctld-86cd78b6f5-zgfqg
++ cat
+ eval exec /vt/bin/vtctld '-cell="zone1"' '-web_dir="/vt/web/vtctld"' '-web_dir2="/vt/web/vtctld2/app"' -workflow_manager_init -workflow_manager_use_election -logtostderr=true -stderrthreshold=0 -port=15000 -grpc_port=15999 '-service_map="grpc-vtctl"' '-topo_implementation="etcd2"' '-topo_global_server_address="etcd-global-client.vitess:2379"' -topo_global_root=/vitess/global
++ exec /vt/bin/vtctld -cell=zone1 -web_dir=/vt/web/vtctld -web_dir2=/vt/web/vtctld2/app -workflow_manager_init -workflow_manager_use_election -logtostderr=true -stderrthreshold=0 -port=15000 -grpc_port=15999 -service_map=grpc-vtctl -topo_implementation=etcd2 -topo_global_server_address=etcd-global-client.vitess:2379 -topo_global_root=/vitess/global
ERROR: logging before flag.Parse: E0422 02:35:34.020928       1 syslogger.go:122] can't connect to syslog
F0422 02:35:39.025400       1 server.go:221] Failed to open topo server (etcd2,etcd-global-client.vitess:2379,/vitess/global): grpc: timed out when dialing

我在1个主节点和2个节点的流浪者后面跑。我怀疑这是eth1的一个问题。

存储配置为使用NFS。

$ kubectl logs etcd-operator-84db9bc774-j4wml
time="2019-04-22T17:26:51Z" level=info msg="skip reconciliation: running ([]), pending ([etcd-zone1-zwgvd7spzc])" cluster-name=etcd-zone1 cluster-namespace=vitess pkg=cluster
time="2019-04-22T17:26:51Z" level=info msg="skip reconciliation: running ([]), pending ([etcd-zone1-zwgvd7spzc])" cluster-name=etcd-global cluster-namespace=vitess pkg=cluster

Answer 1

似乎etcd没有完全初始化。请注意，全局锁定服务器（etcd-global-6pwvnv29th）的pod和cell zone1（pod etcd-zone1-zwgvd7spzc）的本地pod都没有准备好。