WebSphere8.5总是同时挂起线程

Question

我们使用WebSphere AppServer 8.5进行Web处理，但是在特定的时间点(如上午8点)，我们得到线程死锁，最后线程挂起。一小时后，AppServer运行得很慢。然而，一小时后，AppServer恢复了。在检查操作系统的计划任务之后，发现在上午8:00没有执行任何任务。

1CIJAVAVERSION JRE 1.7.0 Windows 7 x86-32 build  (pwi3270sr9-20150417_01(SR9) )
1CIVMVERSION   VM build R26_Java726_SR9_20150406_1443_B242981
1CIJITVERSION  tr.r11_20150401_88894
1CIGCVERSION   GC - R26_Java726_SR9_20150406_1443_B242981
1CIJITMODES    JIT enabled, AOT enabled, FSD disabled, HCR disabled
1CIRUNNINGAS   Running as a standalone JVM
1CIPROCESSID   Process ID: 10332 (0x285C)
1CICMDLINE     C:\IBMRPT_871\SDP\jdk\bin\java.exe -DrptNextgenDebug -Drptserver.rootDiscoveryUrl=http://9.84.122.22:7081/deployment/ -Drptagent.agentName=localhost -Drptagent.engineName=localhost -Drptagent.scheduleName=sched2 -Drptagent.rptDeployDir=file:/C:/Users/IBM_ADMIN/IBM/RPTCitrixTest871/deployment_root/gdaronde/A1E515CFE7BA0690C143E56139636135/ -Drptagent.secure=false -Drptagent.securePort=7444 -Drptagent.installBase=C:/IBMRPT_871/SDP/Majordomo/ -Xmx1200m -DrptLocale=fr_FR -Drptserver.domoHeavyClass=com.ibm.rational.test.lt.kernel.runner.impl.RPTNextgenRunner -Drptserver.domoLiteUrl=file:/C:/IBMRPT_871/SDP/Majordomo/lib/ -Drptserver.domoLiteClass=com.ibm.rational.test.lt.nextgen.Domo -cp C:\IBMRPT_871\SDP\Majordomo\lib\boot.jar com.ibm.rational.test.lt.boot.DomoBooter
1CIJAVAHOMEDIR Java Home Dir:   C:\IBMRPT_871\SDP\jdk\jre
1CIJAVADLLDIR  Java DLL Dir:    C:\IBMRPT_871\SDP\jdk\jre\bin
1CISYSCP       Sys Classpath:   C:\IBMRPT_871\SDP\jdk\jre\bin\default\jclSC170\vm.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\se-service.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\math.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\jlm.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmorb.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmorbapi.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmpkcs.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcertpathfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjgssfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjssefw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmsaslfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjcefw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjgssprovider.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmjsseprovider2.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmcertpathprovider.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\xmldsigfw.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\xml.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\charsets.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\resources.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\rt.jar;C:\IBMRPT_871\SDP\jdk\jre\lib\ibmgpu.jar;
1CIUSERARGS    UserArgs:
2CIUSERARG               -Xoptionsfile=C:\IBMRPT_871\SDP\jdk\jre\bin\default\options.default

0MEMUSER
1MEMUSER       JRE: 1,320,399,028 bytes / 2770 allocations
1MEMUSER       |
2MEMUSER       +--VM: 1,315,550,900 bytes / 2282 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Classes: 16,393,128 bytes / 508 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Memory Manager (GC): 1,284,062,768 bytes / 446 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Java Heap: 1,258,291,200 bytes / 1 allocation
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 25,771,568 bytes / 445 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Threads: 13,546,748 bytes / 236 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Java Stack: 475,728 bytes / 50 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Native Stack: 12,746,752 bytes / 50 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 324,268 bytes / 136 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Trace: 444,968 bytes / 353 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JVMTI: 17,328 bytes / 13 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JNI: 70,504 bytes / 290 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Port Library: 9,632 bytes / 74 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Other: 1,005,824 bytes / 362 allocations
1MEMUSER       |
2MEMUSER       +--JIT: 3,880,280 bytes / 225 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JIT Code Cache: 1,048,576 bytes / 2 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JIT Data Cache: 524,336 bytes / 1 allocation
2MEMUSER       |  |
3MEMUSER       |  +--Other: 2,307,368 bytes / 222 allocations
1MEMUSER       |
2MEMUSER       +--Class Libraries: 967,848 bytes / 263 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Harmony Class Libraries: 1,024 bytes / 1 allocation
2MEMUSER       |  |
3MEMUSER       |  +--VM Class Libraries: 966,824 bytes / 262 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--sun.misc.Unsafe: 72,360 bytes / 10 allocations
4MEMUSER       |  |  |  |
5MEMUSER       |  |  |  +--Direct Byte Buffers: 66,040 bytes / 8 allocations
4MEMUSER       |  |  |  |
5MEMUSER       |  |  |  +--Other: 6,320 bytes / 2 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 894,464 bytes / 252 allocations

3XMTHREADINFO      "Brother-7" J9VMThread:0x518FDD00, j9thread_t:0x527F7634, java/lang/Thread:0x030B1D80, state:P, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x32, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0xA44, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x01020000)
3XMCPUTIME               CPU usage total: 0.0 secs, user: 0.0 secs, system: 0.0 secs
3XMTHREADBLOCK     Parked on: java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x02E83C68 Owned by: <unknown>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at sun/misc/Unsafe.park(Native Method)
4XESTACKTRACE                at java/util/concurrent/locks/LockSupport.park(LockSupport.java:198)
4XESTACKTRACE                at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2055)
4XESTACKTRACE                at java/util/concurrent/LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:501)
4XESTACKTRACE                at java/util/concurrent/LinkedBlockingDeque.take(LinkedBlockingDeque.java:690)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1080)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:798)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               NtWaitForSingleObject+0x15 (0x77A0F8CD [ntdll+0x1f8cd])
4XENATIVESTACK               WaitForSingleObjectEx+0x43 (0x75491194 [kernel32+0x11194])
4XENATIVESTACK               WaitForSingleObject+0x12 (0x75491148 [kernel32+0x11148])
4XENATIVESTACK               j9thread_park+0xba (j9thread.c:2631, 0x73B71C8A [J9THR26+0x1c8a])
4XENATIVESTACK               jclCallThreadPark+0x198 (threadpark.c:82, 0x666C2F88 [jclse7b_26+0x42f88])
4XENATIVESTACK               sun_misc_Unsafe_park+0x5a (smunsafe.asm:14351, 0x666B760A [jclse7b_26+0x3760a])
4XENATIVESTACK               javaProtectedThreadProc+0x9d (vmthread.c:1881, 0x66831DBD [j9vm26+0x51dbd])
4XENATIVESTACK               j9sig_protect+0x44 (j9signal.c:150, 0x6BB9F8B4 [J9PRT26+0xf8b4])
4XENATIVESTACK               javaThreadProc+0x39 (vmthread.c:298, 0x66832739 [j9vm26+0x52739])
4XENATIVESTACK               thread_wrapper+0xda (j9thread.c:1154, 0x73B747AA [J9THR26+0x47aa])
4XENATIVESTACK               _endthread+0x48 (0x67E3C55C [msvcr100+0x5c55c])
4XENATIVESTACK               _endthread+0xe8 (0x67E3C5FC [msvcr100+0x5c5fc])
4XENATIVESTACK               BaseThreadInitThunk+0x12 (0x7549337A [kernel32+0x1337a])
4XENATIVESTACK               RtlInitializeExceptionChain+0x63 (0x77A292B2 [ntdll+0x392b2])
4XENATIVESTACK               RtlInitializeExceptionChain+0x36 (0x77A29285 [ntdll+0x39285])

我们分析了线程信息，发现WebSphere AppServer 8.5和JDK1.7之间可能存在GC冲突，这将在服务器重新启动后得到解决。

Answer 1

我们发现了同样的问题。我们使用Websphere 8.2，在特定的时间点，Websphere被挂起所有线程，并在JVM中造成死锁。因此，我们在IBM上创建了一个案例，他们说Java1.7的完整GC和WebSphere的主线程在某个时候发生了冲突。

我们已经升级到WebSphere 9.0.5和JDK1.8。问题已经解决了。

IBM：https://www.ibm.com/docs/en/ztpf/1.1.0.15?topic=threads-blocked-thread-information

Answer 2

对于john_0xFF，您有关于这个问题的更多细节吗？

我们在WAS8.5上也有同样的问题。我们使用WAS网关，它在上周的16:00到17:00/18:00下降了3次。我们使用Tomcat之后，问题就解决了。