Glassfish 3.12的CPU使用率为100%，有两个用户

Question

我们的玻璃鱼3.12昨晚达到了100%，把用户锁在了系统之外(只有两个)。我转储了玻璃鱼日志(下面)，有大量的锁，虽然没有一个超过1秒的CPU。使用100%的CPU似乎是Glassfish 3.1.2的一个常见问题，但我还没有找到明确的解决方案。任何想法都欢迎。

./asadmin generate-jvm-report --type=thread --user admin


Thread Synchronization Statistics:

-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 12
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
User-level CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-12" thread-id: 24 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4cc6afaa
         at: java.lang.Object.wait(Native Method)
         at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
         at: com.sun.grizzly.Controller.start(Controller.java:869)
         at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
         at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
         at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
         at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
         at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
User-level CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Grizzly-kernel-thread(1)" thread-id: 22 thread-state: RUNNABLE Running in native
         at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
         at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
         at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
         at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
         at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
         at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
         at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
         at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
         at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 1
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
User-level CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Grizzly-kernel-thread(1)" thread-id: 20 thread-state: RUNNABLE Running in native
         at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
         at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
         at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
         at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
         at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
         at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
         at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
         at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
         at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 3
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
User-level CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-7" thread-id: 19 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@836e947
         at: java.lang.Object.wait(Native Method)
         at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
         at: com.sun.grizzly.Controller.start(Controller.java:869)
         at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
         at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
         at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
         at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
         at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
User-level CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-4" thread-id: 16 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4d0d3887
         at: java.lang.Object.wait(Native Method)
         at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
         at: com.sun.grizzly.Controller.start(Controller.java:869)
         at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
         at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
         at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
         at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
         at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 2
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
User-level CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-2" thread-id: 14 thread-state: WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1efa4e96
         at: sun.misc.Unsafe.park(Native Method)
         at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
         at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
         at: com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:665)
         at: com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:179)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 78
Total CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
User-level CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "pool-1-thread-1" thread-id: 13 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@433d69bd
         at: sun.misc.Unsafe.park(Native Method)
         at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
         at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
         at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
         at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
         at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
         at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 1,892
Total CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
User-level CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "FelixStartLevel" thread-id: 12 thread-state: WAITING Waiting on lock: java.util.ArrayList@74e919c7
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:503)
         at: org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:272)
         at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 42
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 42
Total CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
User-level CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "FelixDispatchQueue" thread-id: 11 thread-state: WAITING Waiting on lock: java.util.ArrayList@739b93c7
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:503)
         at: org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
         at: org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
         at: org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
         at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 232
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 217
Total CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
User-level CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Signal Dispatcher" thread-id: 4 thread-state: RUNNABLE
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 70,212 nanoseconds.
User-level CPU time for this thread: 0 seconds 70,212 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Finalizer" thread-id: 3 thread-state: WAITING Waiting on lock: java.lang.ref.ReferenceQueue$Lock@6b03291c
         at: java.lang.Object.wait(Native Method)
         at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
         at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
         at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 20
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 19
Total CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
User-level CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Reference Handler" thread-id: 2 thread-state: WAITING Waiting on lock: java.lang.ref.Reference$Lock@1c802427
         at: java.lang.Object.wait(Native Method)
         at: java.lang.Object.wait(Object.java:503)
         at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 18
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 18
Total CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
User-level CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
No deadlock found

Answer 1

我在日志中发现了由于垃圾收集而产生的OutOfMemory。我增加了Glassfish服务器中的一些JVM参数，并将其设置为" server“而不是"client”，这似乎还可以。

Answer 2

你可能会发现一些有用的东西：

glassfish full gc once an hour

01/html/821-2431/abeic.html