Thursday, 22 August 2013

Hundreds of JBoss 5.1 System Threads WAITING forever

Hundreds of JBoss 5.1 System Threads WAITING forever

About 3 months ago we migrated to JBoss 5.1.0.GA and a few days after
testing we found several Out Of Memory Errors (OOME) in the logs of our
app.
Then I found an issue that described the cause as a bug in the JVM up to
version 7. We updated to version 7u25 and we haven't seen more OOME but
now I see an abnormally huge thread count: about 2k threads and of them
1.9k are daemon threads.
After checking in our monitoring tools I discovered that they are all
threads generated by the JBoss system thread pool (they are all named
JBoss System Threads(1)-XXXX). And here is the Stack Trace Details:
"JBoss System Threads(1)-1649" Id=130217 in WAITING on
lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a4d1d3
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
I have also checked in the tool for collection leaks and found the next
data that I think is somewhat related:

I have checked JBoss JIRA but haven't found something related.
Can someone point me what is going on?
UPDATE:
Here is the thread pool configuration in the jboss-service.xml
<!-- A Thread pool service -->
<mbean code="org.jboss.util.threadpool.BasicThreadPool"
name="jboss.system:service=ThreadPool">
<attribute name="Name">JBoss System Threads</attribute>
<attribute name="ThreadGroupName">System Threads</attribute>
<!-- How long a thread will live without any tasks in MS -->
<attribute name="KeepAliveTime">60000</attribute>
<!-- The max number of threads in the pool -->
<attribute name="MaximumPoolSize">3200</attribute>
<!-- The max number of tasks before the queue is full -->
<attribute name="MaximumQueueSize">3200</attribute>
<!-- The behavior of the pool when a task is added and the queue is
full.
abort - a RuntimeException is thrown
run - the calling thread executes the task
wait - the calling thread blocks until the queue has room
discard - the task is silently discarded without being run
discardOldest - check to see if a task is about to complete and enque
the new task if possible, else run the task in the calling thread
-->
<attribute name="BlockingMode">run</attribute>
</mbean>

No comments:

Post a Comment