我们在生产中部署了 WSO2 API Manager 2.2.0。
我们的架构: 我们在 DMZ 中部署了一个 API 网关实例。此 API 网关实例通过负载均衡器将所有有效 API 请求转发到内部“一体化”WSO2 API Manager 实例(如果 Prod 实例不可用,请求将路由到 DR 实例)。该一体化实例使用 Oracle 数据库进行 API、注册表、用户管理、消息代理和统计。
我们的日志中每小时都会发生一个错误。此错误的影响似乎是 API 管理器在大约 1 分钟内无法与端点通信,直到问题似乎自行解决。因此,WSO2 将向客户端回复“发件人错误”消息,并且 API 将暂停 30 秒。
谁能建议这个问题的原因可能是什么?
每小时发生的错误:
TID: [-1] [] [2019-01-30 01:00:12,156] WARN {org.wso2.andes.client.protocol.AMQProtocolHandler} - Timed out while waiting for heartbeat from peer. {org.wso2.andes.client.protocol.AMQProtocolHandler}
TID: [-1] [] [2019-01-30 01:00:12,272] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,280] ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina {org.wso2.andes.transport.network.mina.MinaNetworkHandler}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:748)
TID: [-1] [] [2019-01-30 01:00:12,519] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:49069(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,520] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '246' on channel [/<ip-address>:49069(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,456] ERROR {org.wso2.andes.server.protocol.AMQProtocolEngine} - IOException caught in/<ip-address>:7076(user), session closed implictly: java.io.IOException: Connection reset by peer {org.wso2.andes.server.protocol.AMQProtocolEngine}
TID: [-1] [] [2019-01-30 01:00:12,522] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,526] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '44' on channel [/<ip-address>:7076(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,530] ERROR {org.wso2.andes.client.state.AMQStateManager} - No Waiters for error saving as last error:Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. {org.wso2.andes.client.state.AMQStateManager}
TID: [-1] [] [2019-01-30 01:00:12,533] ERROR {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - JMS Connection failed : Exception thrown against AMQConnection:
Host: <ip-address>
Port: 5672
Virtual Host: carbon
Client ID: clientid
Active session count: 1: org.wso2.andes.AMQDisconnectedException: Server closed connection and reconnection not permitted. - shutting down worker tasks {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,534] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt : 1 for Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:12,536] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing all consumers on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,549] INFO {org.wso2.andes.server.AMQChannel} - Unsubscribing consumer '1' on channel [/<ip-address>:29986(user):1] {org.wso2.andes.server.AMQChannel}
TID: [-1] [] [2019-01-30 01:00:12,552] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=386@NODE<server>/<ip-address>/T=1548808789236/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=385@<server>/<ip-address>/T=1548808788762/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,557] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,773] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:12,780] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local Subscription DELETED [throttleData]ID=384@<server>/<ip-address>/T=1548806362449/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=false {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:00:12,782] INFO {org.wso2.andes.kernel.OrphanedMessageHandler} - Purging messages of this node persisted under throttleData {org.wso2.andes.kernel.OrphanedMessageHandler}
TID: [-1] [] [2019-01-30 01:00:12,791] INFO {org.wso2.andes.kernel.MessagingEngine} - Purged messages of destination throttleData {org.wso2.andes.kernel.MessagingEngine}
TID: [-1] [] [2019-01-30 01:00:13,517] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:49069) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,751] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:29986) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:13,926] INFO {org.wso2.andes.kernel.FlowControlManager} - Channel removed (ID: <ip-address>:7076) {org.wso2.andes.kernel.FlowControlManager}
TID: [-1] [] [2019-01-30 01:00:17,607] WARN {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Unable to shutdown all polling tasks of Siddhi-JMS-Consumer {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,609] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for jms consumer 1000 shutdown {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:17,612] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Task manager for Siddhi-JMS-Consumer [re-]initialized {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - SASL Mechanism selected: PLAIN {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,760] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Locale selected: en_US {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,782] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Connected as: user{org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,783] INFO {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler} - Framesize set to 65535 {org.wso2.andes.server.handler.ConnectionStartOkMethodHandler}
TID: [-1234] [] [2019-01-30 01:00:17,873] INFO {org.wso2.andes.server.handler.ChannelOpenHandler} - Connecting to: carbon {org.wso2.andes.server.handler.ChannelOpenHandler}
TID: [-1234] [] [2019-01-30 01:00:17,875] INFO {org.wso2.andes.kernel.AndesChannel} - Channel created (ID: <ip-address>:31136) {org.wso2.andes.kernel.AndesChannel}
TID: [-1234] [] [2019-01-30 01:00:17,957] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 bound to default exchange(<<default>>) {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:17,958] INFO {org.wso2.andes.server.handler.QueueDeclareHandler} - Queue tmp_ip-address_31136_1 declared successfully {org.wso2.andes.server.handler.QueueDeclareHandler}
TID: [-1234] [] [2019-01-30 01:00:18,074] INFO {org.wso2.andes.server.handler.QueueBindHandler} - Binding queue tmp_ip-address_31136_1 to exchange TopicExchange[amq.topic] with routing key throttleData {org.wso2.andes.server.handler.QueueBindHandler}
TID: [-1] [] [2019-01-30 01:00:18,615] INFO {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager} - Reconnection attempt: 1 for Siddhi-JMS-Consumer was successful! {org.wso2.carbon.apimgt.jms.listener.utils.JMSTaskManager}
TID: [-1] [] [2019-01-30 01:00:20,428] INFO {org.wso2.andes.subscription.SubscriptionEngine} - Local subscription ADDED [throttleData]ID=387@<server>/<ip-address>/T=1548810020424/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@6b424af0/EUD=0/S=true {org.wso2.andes.subscription.SubscriptionEngine}
TID: [-1] [] [2019-01-30 01:05:28,619] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
事实是 wso2 使用
Apache Mina
来管理与 JMS 的通信。
WSO2 使用 JMS 创建请求队列。您可能正在向无法访问的 API 发出请求。
因为这对我来说就是问题。
有时问题可能出在 JMS 及其连接上。如果您仍然感兴趣,请在此处向我展示日志文件