Wildfli10.1.0最终远程处理端点任务线程不断增长

Question

我们使用的是野蝇10.1.0决赛。

我们遇到了一个由不断增长的线程引起的OutOfMemoryError。

在检查线程转储之后。

我们发现有数千个远程处理“端点”任务-N个线程。

远程处理“终结点”任务是什么-N个线程？

它们是由jobss-remoting创建的吗

在重新启动服务器之后，我们发现在启动过程中，只有16个线程：

远程处理“端点”任务-1~远程处理“终结点”任务-16。

服务器运行服务器数天或数月后，可能会有数百个或数千个远程线程：

下面列出了线程转储的片段。

在这个线程转储中，有几个"Remoting“端点”-11“的不同编号。

其他任务也是如此，比如任务-1到任务-16。

所有这些线程都在等待。

"Remoting "endpoint" task-11" #55415 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a8000 nid=0x276e waiting on condition [0x00007f280a36c000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-11" #55417 daemon prio=5 os_prio=0 tid=0x00007f2ba003f800 nid=0x276d waiting on condition [0x00007f2794818000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-11" #55414 daemon prio=5 os_prio=0 tid=0x00007f2b98023800 nid=0x276b waiting on condition [0x00007f2792bfc000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-10" #55411 daemon prio=5 os_prio=0 tid=0x00007f2ba003e000 nid=0x276a waiting on condition [0x00007f27926f7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabecf40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-10" #55413 daemon prio=5 os_prio=0 tid=0x00007f2b8c0a7000 nid=0x2769 waiting on condition [0x00007f27927f8000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabee2b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-10" #55412 daemon prio=5 os_prio=0 tid=0x00007f2b98022800 nid=0x2768 waiting on condition [0x00007f27c4815000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabeda50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-9" #55372 daemon prio=5 os_prio=0 tid=0x00007f2c7408f000 nid=0x41df waiting on condition [0x00007f27907d8000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-8" #55369 daemon prio=5 os_prio=0 tid=0x00007f2c7408d000 nid=0x41dd waiting on condition [0x00007f27909da000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-7" #55368 daemon prio=5 os_prio=0 tid=0x00007f2c7408b000 nid=0x41dc waiting on condition [0x00007f2790adb000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-6" #55367 daemon prio=5 os_prio=0 tid=0x00007f2c74089000 nid=0x41db waiting on condition [0x00007f2790bdc000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-5" #55366 daemon prio=5 os_prio=0 tid=0x00007f2c74087000 nid=0x41da waiting on condition [0x00007f2790cdd000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-4" #55365 daemon prio=5 os_prio=0 tid=0x00007f2c74085000 nid=0x41d9 waiting on condition [0x00007f2790dde000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabece88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-9" #55364 daemon prio=5 os_prio=0 tid=0x00007f2bd813c000 nid=0x41d8 waiting on condition [0x00007f2790edf000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabed500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


"Remoting "endpoint" task-9" #55363 daemon prio=5 os_prio=0 tid=0x00007f2bf4044000 nid=0x41d7 waiting on condition [0x00007f2790fe0000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006eabee3c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

....

20180903

我发现"Remoting“端点”任务“线程是由"xnio”创建的。我发现xnio有一个与我们的场景非常相似的问题：

https://issues.jboss.org/browse/XNIO-285

它说这个问题已经在"xnio 3.6.0.Beta 1“中解决了。不幸的是，Wildflix10.1.0使用的是xnio3.4.0。当我试图升级到xnio3.6.5时，我得到了org/ java.lang.NoClassDefFoundError /通配性/上下文/上下文。在升级了通配符-公共-1.4.0.Final.jar(包含“org/通配符/公共/上下文/上下文”类)之后，NoClassDefFoundError仍然存在。

还有其他方法来阻止远程处理“端点”任务线程的增长吗？

Answer 1

您可能正在使用scoped Execution执行远程方法。

每个作用域的EJB上下文都将创建新线程，简单地调用context.close()方法不会关闭上下文，因此您将获得OutOfMemoryError

如何关闭作用域的EJB客户机上下文？

答案是相同的，在EJB客户机上下文中使用close()方法。但真正的问题是如何获得与JNDI上下文相关联的相关作用域EJB客户端上下文。在讨论这一点之前，了解用于EJB查找的ejb: JNDI命名空间以及JNDI上下文(通常是您在客户端代码中看到的InitialContext )是如何关联的非常重要。Java语言提供的JNDI允许在JNDI框架中注册"URL上下文工厂“(详见http://docs.oracle.com/javase/jndi/tutorial/provider/url/factory.html)。与文档声明一样，在JNDI查找期间，可以使用URL上下文工厂解析URL字符串。这就是当您进行远程EJB查找时的ejb:前缀。ejb: URL字符串由URL上下文工厂支持。

在内部，当对ejb: URL字符串进行查找时，将为该ejb: javax.naming.Context创建相关的javax.naming.Context。让我们看看一些代码，以便更好地理解：

// JNDI context "A"
Context jndiCtx = new InitialContext(props);
// Now let's lookup a EJB
MyBean bean = jndiCtx.lookup("ejb:app/module/distinct/bean!interface");

因此，我们首先创建一个JNDI上下文，然后使用它查找EJB。虽然，使用ejb: JNDI名称的bean查找只是一条语句，但它涉及到更多的内容。在查找该字符串时，实际发生的情况是为ejb: URL字符串创建了一个单独的javax.naming.Context。然后，这个新的javax.naming.Context用于查找JNDI名称中的其余字符串。

让我们将这一行分解成多个语句，以便更好地理解：

// Remember, the ejb: is backed by a URL context factory which returns a Context for the ejb: URL (that's why it's called a context factory)
final Context ejbNamingContext = (Context) jndiCtx.lookup("ejb:");
// Use the returned EJB naming context to lookup the rest of the JNDI string for EJB
final MyBean bean = ejbNamingContext.lookup("app/module/distinct/bean!interface");

正如您在上面看到的，我们将这个单一的语句分成几个语句，以便更好地解释细节。因此，正如您在JNDI名称中解析ejb: URL字符串时所看到的那样，它获得了一个javax.naming.Context实例。此实例不同于用于查找的实例(本例中为jndiCtx)。这是一个需要理解的重要细节(基于稍后解释的原因)。现在，这个返回的实例用于查找JNDI字符串的其余部分(“app/模块/distinct/bean！接口”)，然后返回EJB代理。不管查找是在一个语句中完成还是在多个部分中完成，代码的工作方式都是相同的。例如，为ejb: URL字符串创建javax.naming.Context实例。

那么，当这个部分的标题为“如何关闭范围内的EJB客户端上下文”时，我为什么要解释这些呢？原因是，处理与JNDI上下文关联的作用域EJB客户端上下文的客户端应用程序将期望以下代码关闭关联的EJB客户端上下文，但会感到惊讶的是，它不会：

final Properties props = new Properties();
// mark it for scoped EJB client context
props.put("org.jboss.ejb.client.scoped.context","true");
// add other properties
props.put(....);
...
Context jndiCtx = new InitialContext(props);
try {
      final MyBean bean = jndiCtx.lookup("ejb:app/module/distinct/bean!interface");
      bean.doSomething();
} finally {
  jndiCtx.close();
}

应用程序期望对jndiCtx.close()的调用将有效地关闭与JNDI上下文关联的EJB上下文。这种情况不会发生，因为如前所述，支持ejb: URL字符串的javax.naming.Context与代码正在关闭的实例不同。Java中的JNDI实现，只是关闭了调用关闭的上下文。因此，支持ejb: URL字符串的另一个javax.naming.Context仍然没有关闭，这实际上意味着作用域EJB客户端上下文也没有关闭，这最终意味着到EJB客户机上下文中的服务器的连接也没有关闭。

现在，让我们看看如何正确地完成这一任务。我们知道ejb: URL字符串查找返回一个javax.naming.Context。我们所要做的就是保持对这个实例的引用，并在完成EJB调用时关闭它。下面是它的样子：

final Properties props = new Properties();
// mark it for scoped EJB client context
props.put("org.jboss.ejb.client.scoped.context","true");
// add other properties
props.put(....);
...
Context jndiCtx = new InitialContext(props);
Context ejbRootNamingContext = (Context) jndiCtx.lookup("ejb:");
try {
    final MyBean bean = ejbRootNamingContext.lookup("app/module/distinct/bean!interface"); // the rest of the EJB jndi string
    bean.doSomething();
} finally {
    try {
        // close the EJB naming JNDI context
        ejbRootNamingContext.close();
    } catch (Throwable t) {
        // log and ignore
    }
    try {
        // also close our other JNDI context since we are done with it too
        jndiCtx.close();
    } catch (Throwable t) {
        // log and ignore
    }

}

正如您所看到的，我们将代码更改为首先对" EJB :“字符串进行查找，以获取EJB命名上下文，然后使用该ejbRootNamingContext实例查找EJB名称的其余部分以获取EJB代理。然后，当关闭上下文时，我们关闭了ejbRootNamingContext (以及其他JNDI上下文)。关闭ejbRootNamingContext确保与该JNDI上下文关联的作用域EJB上下文也被关闭。实际上，这将关闭EJB客户端上下文中与服务器的连接。

有关更多详细信息，请参阅作用域EJB客户端上下文

Answer 2

我发现javax.management.remote.JMXConnector.创建了远程处理“终结点”任务线程我们打开一些javax.management.remote.JMXConnector来访问其他服务器中的MBeans。但并没有关闭它们。关闭这些JMXConnector实例之后，线程就消失了。

javax.management.remote.JMXConnector正在使用xnio与MBeans通信。它将在打开时创建一个XnioWorker，XnioWorker将创建远程处理“终结点”任务线程。因此，这个问题不是由EJB引起的。