WebSphere spring-websocket clientOutboundChannelExecutor死锁

Question

我们正在使用spring-websocket和WebSphere 16.0.0.3，当所有的clientOutboundChannelExecutor都被卡住时，我们一次又一次地面临着问题。

3XMTHREADINFO      "clientOutboundChannel-1" J9VMThread:0x0000000001834900, j9thread_t:0x00007F2EDEFD1030, java/lang/Thread:0x00000000E2E30A28, state:B, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x44, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0xDF6A, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
3XMTHREADINFO2            (native stack address range from:0x00007F2EDE11A000, to:0x00007F2EDE15B000, size:0x41000)
3XMCPUTIME               CPU usage total: 49.720264588 secs, current category="Application"
3XMTHREADBLOCK     Blocked on: java/lang/Object@0x00000000E30B30F8 Owned by: "clientOutboundChannel-4" (J9VMThread:0x00000000019CEE00, java/lang/Thread:0x00000000E2E30C98)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.resetRequest(AbstractHttpSockJsSession.java:334(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.disconnect(AbstractHttpSockJsSession.java:330(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.close(AbstractSockJsSession.java:216(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:158(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:192(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:457(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))

3XMTHREADINFO      "clientOutboundChannel-2" J9VMThread:0x00000000019A5A00, j9thread_t:0x00007F2EDEFD15B0, java/lang/Thread:0x00000000E2E30AF8, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x45, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0xDF6B, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x01000101)
3XMTHREADINFO2            (native stack address range from:0x00007F2EDDE5F000, to:0x00007F2EDDEA0000, size:0x41000)
3XMCPUTIME               CPU usage total: 0.281253777 secs, current category="Application"
3XMTHREADBLOCK     Waiting on: com/ibm/ws/tcpchannel/internal/SimpleSync@0x00000000E3155FC8 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Object.wait(Native Method)
4XESTACKTRACE                at java/lang/Object.wait(Object.java:172)
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/SimpleSync.simpleWait(SimpleSync.java:35)
5XESTACKTRACE                   (entered lock: com/ibm/ws/tcpchannel/internal/SimpleSync@0x00000000E3155FC8, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/NioTCPWriteRequestContextImpl.processSyncWriteRequest(NioTCPWriteRequestContextImpl.java:77(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:119(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.synchWrite(HttpServiceContextImpl.java:2438(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2174(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/inbound/HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:753(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flushBuffers(HttpOutputStreamImpl.java:481(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flush(HttpOutputStreamImpl.java:579(Compiled Code))
4XESTACKTRACE                at com/ibm/wsspi/http/ee7/HttpOutputStreamEE7.flush(HttpOutputStreamEE7.java:293(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/response/WCOutputStream.flush(WCOutputStream.java:234(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer31/osgi/response/WCOutputStream31.flush(WCOutputStream31.java:599(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:705(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:670)
4XESTACKTRACE                at org/springframework/http/server/ServletServerHttpResponse.flush(ServletServerHttpResponse.java:96(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.writeFrameInternal(AbstractHttpSockJsSession.java:365)
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.writeFrame(AbstractSockJsSession.java:322(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/StreamingSockJsSession.flushCache(StreamingSockJsSession.java:86(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.sendMessageInternal(AbstractHttpSockJsSession.java:302(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x00000000E33995A8, entry count: 1)
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.sendMessage(AbstractSockJsSession.java:166(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.tryFlushMessageBuffer(ConcurrentWebSocketSessionDecorator.java:132(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.sendMessage(ConcurrentWebSocketSessionDecorator.java:104(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:440(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))

3XMTHREADINFO      "clientOutboundChannel-3" J9VMThread:0x00000000019C9D00, j9thread_t:0x00007F2EDEFDAE00, java/lang/Thread:0x00000000E2E30BC8, state:B, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x46, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0xDF6C, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
3XMTHREADINFO2            (native stack address range from:0x00007F2EDDE1E000, to:0x00007F2EDDE5F000, size:0x41000)
3XMCPUTIME               CPU usage total: 42.712892035 secs, current category="Application"
3XMTHREADBLOCK     Blocked on: java/lang/Object@0x00000000E33995A8 Owned by: "clientOutboundChannel-2" (J9VMThread:0x00000000019A5A00, java/lang/Thread:0x00000000E2E30AF8)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.resetRequest(AbstractHttpSockJsSession.java:334(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.disconnect(AbstractHttpSockJsSession.java:330)
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.close(AbstractSockJsSession.java:216(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:158)
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:192)
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:457(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))

3XMTHREADINFO      "clientOutboundChannel-4" J9VMThread:0x00000000019CEE00, j9thread_t:0x00007F2EDEFDB380, java/lang/Thread:0x00000000E2E30C98, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x47, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0xDF6D, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x01000101)
3XMTHREADINFO2            (native stack address range from:0x00007F2EDDDDD000, to:0x00007F2EDDE1E000, size:0x41000)
3XMCPUTIME               CPU usage total: 7.811379586 secs, current category="Application"
3XMTHREADBLOCK     Waiting on: com/ibm/ws/tcpchannel/internal/SimpleSync@0x00000000E30B31E8 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Object.wait(Native Method)
4XESTACKTRACE                at java/lang/Object.wait(Object.java:172(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/SimpleSync.simpleWait(SimpleSync.java:35(Compiled Code))
5XESTACKTRACE                   (entered lock: com/ibm/ws/tcpchannel/internal/SimpleSync@0x00000000E30B31E8, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/NioTCPWriteRequestContextImpl.processSyncWriteRequest(NioTCPWriteRequestContextImpl.java:77(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/tcpchannel/internal/TCPWriteRequestContextImpl.write(TCPWriteRequestContextImpl.java:119(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.synchWrite(HttpServiceContextImpl.java:2438(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/HttpServiceContextImpl.sendOutgoing(HttpServiceContextImpl.java:2174(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/inbound/HttpInboundServiceContextImpl.sendResponseBody(HttpInboundServiceContextImpl.java:753(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flushBuffers(HttpOutputStreamImpl.java:481(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/http/channel/internal/outbound/HttpOutputStreamImpl.flush(HttpOutputStreamImpl.java:579(Compiled Code))
4XESTACKTRACE                at com/ibm/wsspi/http/ee7/HttpOutputStreamEE7.flush(HttpOutputStreamEE7.java:293(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/response/WCOutputStream.flush(WCOutputStream.java:234(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer31/osgi/response/WCOutputStream31.flush(WCOutputStream31.java:599(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:705(Compiled Code))
4XESTACKTRACE                at com/ibm/ws/webcontainer/srt/SRTServletResponse.flushBuffer(SRTServletResponse.java:670(Compiled Code))
4XESTACKTRACE                at org/springframework/http/server/ServletServerHttpResponse.flush(ServletServerHttpResponse.java:96(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.writeFrameInternal(AbstractHttpSockJsSession.java:365(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.writeFrame(AbstractSockJsSession.java:322(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/StreamingSockJsSession.flushCache(StreamingSockJsSession.java:86(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractHttpSockJsSession.sendMessageInternal(AbstractHttpSockJsSession.java:302(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/Object@0x00000000E30B30F8, entry count: 1)
4XESTACKTRACE                at org/springframework/web/socket/sockjs/transport/session/AbstractSockJsSession.sendMessage(AbstractSockJsSession.java:166(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.tryFlushMessageBuffer(ConcurrentWebSocketSessionDecorator.java:132(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/handler/ConcurrentWebSocketSessionDecorator.sendMessage(ConcurrentWebSocketSessionDecorator.java:104(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.sendToClient(StompSubProtocolHandler.java:440(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/StompSubProtocolHandler.handleMessageToClient(StompSubProtocolHandler.java:427(Compiled Code))
4XESTACKTRACE                at org/springframework/web/socket/messaging/SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:338(Compiled Code))

IBMSpringApplicationServer16.0.0.3 (1.0.14.cl160320160831-1555)，版本为pxa6480sr3fp11-20160817_02 (SR3 FP11) (en_US) WebSphere -websocket 4.3.3.RELEASE

看起来像是SRTServletResponse.flushBuffer实现可以无限地锁定线程，而SimpleSync.simpleWait永远不会释放线程。我尝试设置tcpOptions inactivityTimeout，但没有帮助。

WebSphere错误？

Answer 1

我们也有同样的问题。目前还没有解决方法或修复方法。

详细信息：

• Java 8.0.65
• Windows Server 2012 R2 x64
• 春季启动1.3.6.RELEASE
• Spring Websocket 4.2.6.RELEASE
• Undertow 1.3.22.Final
• XNIO 3.3.6最终的