Jenkins、MiniKube 和 Kubernetes 插件 - 动态生成代理

问题描述 投票：0回答：1

问题

我已将 csanchez 的 jenkins-kubernetes 插件（版本 0.12）部署到本地 Minikube / Kubernetes 环境。从 Jenkins UI 中配置 Pod 模板和容器模板时，PODS 会自动生成并处理简单的作业。但是，当在 pipeline 脚本中定义 Pod-Templates 和 Container-Templates 时，Jenkins 控制器会拒绝连接，并表示 POD 已连接到控制器。

环境

minikube 版本：

v0.20.0

Kubernetes 版本：

v1.7.1

Jenkins 控制器：

jenkins:2.74-alpine
（Docker 容器）

Jenkins 代理：

jnlp-slave:3.10-1-alpine
（Docker 容器）

操作系统：

4.8.0-59-generic #64-Ubuntu SMP Thu Jun 29 19:38:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

管道线脚本

podTemplate(name: 'my-pod-template', label: 'maven', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'my-container-template', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]
) {

  node('maven') {
    stage('Build a Maven project') {
      container('maven') {
          sh 'sleep 10'
      }
    }
  }
}

生成的 PODS kubectl get pods --all-namespaces -w

NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
kube-system   kube-addon-manager-minikube   1/1       Running   9          7d
kube-system   kube-dns-1301475494-zp3tg     3/3       Running   27         7d
kube-system   kubernetes-dashboard-jlsk7    1/1       Running   9          7d
mmidevops     jenkins-83791151-msdcw        1/1       Running   30         6d
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       ContainerCreating   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   2/2       Running   0         6s
mmidevops   my-pod-template-nfdgh-9k7qh   1/2       Error     0         9s

来自代理 POD 的日志

INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:20 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:20 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to 172.17.0.2/172.17.0.2:50000] Local headers refused by remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:377)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
    ... 7 more

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP2-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)

Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:644)
at hudson.remoting.Engine.innerRun(Engine.java:608)
at hudson.remoting.Engine.run(Engine.java:447)

来自 Jenkins Master 的日志

Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template: Kubernetes Pod Template
Aug 28, 2017 11:35:14 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #38 from /172.17.0.5:54934
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #39 from /172.17.0.5:54944
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection from 172.17.0.5/172.17.0.5:54944] Refusing headers from remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP2-connect connection #40 from /172.17.0.5:54960
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP-connect connection #41 from /172.17.0.5:54962
Aug 28, 2017 11:35:23 AM hudson.node_monitors.ResponseTimeMonitor$1 monitor
WARNING: Making my-pod-template-qdgzl-29t7l offline because it’s not responding
Aug 28, 2017 11:35:24 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 7 computer(s)
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer my-pod-template-nfdgh-9k7qh
Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#41] for my-pod-template-nfdgh-9k7qh terminated
java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
    at hudson.remoting.Channel.close(Channel.java:1304)
    at hudson.remoting.Channel.close(Channel.java:1272)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Aug 28, 2017 11:40:20 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TestPipeLine #94 completed: FAILURE

JNLP 日志

Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:15 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:15 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

更新 因此，经过更多谷歌搜索后，我发现了这个......“如果您在 Pod 定义中定义了 JNLP 容器，则需要将其删除或重命名为jnlp，否则将创建一个名为 jnlp 的新容器” ...这就是说...

podTemplate(name: 'default-java-slave', label: 'maven-builder', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'jnlp', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: 
'/var/run/docker.sock')
]
) {

  node('maven-builder') {
    stage('Build a Maven project') {
      container('jnlp') {
          sh 'sleep 10'
      }
    }
  }
}
kubernetes jenkins minikube jnlp jenkins-agent
1个回答
1
投票

我使用 kubernetes 插件 0.12 并在 Jenkins 代理上遇到 ClosedChannelException 问题（我认为这不是连接到 master 的问题，但通信不可接受，然后连接超时并关闭），我在升级 Jenkins 版本时解决了这个问题从 2.60-alpine 到 2.75-apline。詹金斯代理现在工作正常。希望这能帮助您解决问题。

PS。 Kubernetes 插件已经发布了 1.0 的新版本，我也升级到了新版本，它也运行得很好！

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.