探讨基于java的web应用性能随时间的下降策略

Question

我正在研究企业级java应用程序，它已经包含了许多工具/框架，比如Struts、JAX和Spring。它包含.war文件中捆绑在一起的UI和REST端点。这个项目正在发展，我们正在摆脱旧的工具，努力坚持只使用Spring /Webflux。

应用程序正在对数百万条XML/JSON记录执行搜索，最近搜索引擎从Marklogic切换到Elasticsearch。

我们已经注意到的是，在生产中，一些端点的响应时间随着时间的推移而增加，而不是大量使用( 2-4个应用程序节点的响应时间高达1.7k rpm )。Elasticsearch有增长的空间，没有任何巨大负载的迹象。因此，目前我们必须重新启动/替换prod实例一周或两周，当平均响应时间超过3秒时，而不是普通的200-300Millis。

我试着使用异步分析器获取CPU和堆火焰图，但是负载配置文件在每个度量上都在变化，因为我们有很多可用的特性，所以我无法比较图表是如何随时间变化的。

你能告诉我在代码中找到合适位置的一些策略/方法吗？

Answer 1

发现问题了。它与线程池相关。

我们注意到，随着时间的推移，活动tomcat线程的数量与响应时间一起增长：

在图像中，您还可以看到服务器在5月9日重新启动。

在服务器重新启动之前，我能够获得一个堆转储，在挖掘之后，我在线程转储中发现了一个有趣的重复的片段：

Thread xxx
  at sun.misc.Unsafe.park(ZJ)V (Native Method)
  at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()V (AbstractQueuedSynchronizer.java:2039)
  at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(Ljava/lang/Object;Ljava/lang/Object;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/Future;)Lorg/apache/http/pool/PoolEntry; (AbstractConnPool.java:377)
  at org.apache.http.pool.AbstractConnPool.access$200(Lorg/apache/http/pool/AbstractConnPool;Ljava/lang/Object;Ljava/lang/Object;JLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/Future;)Lorg/apache/http/pool/PoolEntry; (AbstractConnPool.java:67)
  at org.apache.http.pool.AbstractConnPool$2.get(JLjava/util/concurrent/TimeUnit;)Lorg/apache/http/pool/PoolEntry; (AbstractConnPool.java:243)
  at org.apache.http.pool.AbstractConnPool$2.get(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object; (AbstractConnPool.java:191)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(Ljava/util/concurrent/Future;JLjava/util/concurrent/TimeUnit;)Lorg/apache/http/HttpClientConnection; (PoolingHttpClientConnectionManager.java:282)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(JLjava/util/concurrent/TimeUnit;)Lorg/apache/http/HttpClientConnection; (PoolingHttpClientConnectionManager.java:269)
  at org.apache.http.impl.execchain.MainClientExec.execute(Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/client/methods/HttpRequestWrapper;Lorg/apache/http/client/protocol/HttpClientContext;Lorg/apache/http/client/methods/HttpExecutionAware;)Lorg/apache/http/client/methods/CloseableHttpResponse; (MainClientExec.java:191)
  at org.apache.http.impl.execchain.ProtocolExec.execute(Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/client/methods/HttpRequestWrapper;Lorg/apache/http/client/protocol/HttpClientContext;Lorg/apache/http/client/methods/HttpExecutionAware;)Lorg/apache/http/client/methods/CloseableHttpResponse; (ProtocolExec.java:185)
  at org.apache.http.impl.execchain.RetryExec.execute(Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/client/methods/HttpRequestWrapper;Lorg/apache/http/client/protocol/HttpClientContext;Lorg/apache/http/client/methods/HttpExecutionAware;)Lorg/apache/http/client/methods/CloseableHttpResponse; (RetryExec.java:89)
  at org.apache.http.impl.execchain.RedirectExec.execute(Lorg/apache/http/conn/routing/HttpRoute;Lorg/apache/http/client/methods/HttpRequestWrapper;Lorg/apache/http/client/protocol/HttpClientContext;Lorg/apache/http/client/methods/HttpExecutionAware;)Lorg/apache/http/client/methods/CloseableHttpResponse; (RedirectExec.java:111)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(Lorg/apache/http/HttpHost;Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse; (InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/client/methods/CloseableHttpResponse; (CloseableHttpClient.java:83)
  at org.apache.http.impl.client.CloseableHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; (CloseableHttpClient.java:108)
  at io.searchbox.client.http.JestHttpClient.executeRequest(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; (JestHttpClient.java:136)
  at io.searchbox.client.http.JestHttpClient.execute(Lio/searchbox/action/Action;Lorg/apache/http/client/config/RequestConfig;)Lio/searchbox/client/JestResult; (JestHttpClient.java:70)
  at io.searchbox.client.http.JestHttpClient.execute(Lio/searchbox/action/Action;)Lio/searchbox/client/JestResult; (JestHttpClient.java:63)
...

在我们的例子中，我们使用Jest库与Elasticsearch对话。在内部，它使用Apache客户端& Apache异步客户端。

正如您在线程转储中所看到的，很明显，该线程正在HTTP客户端的线程池中等待可用线程。而且有更多的线程与完全相同的堆栈。

我还发现，我们将maxTotal (最大连接总数)设置为20，将defaultMaxPerRoute (每个路由的最大连接)设置为2。

默认情况下，池总共只允许20个并发连接，每个唯一路由只允许两个并发连接。两个连接限制是由于HTTP规范的要求造成的。然而，在实际条件下，这往往是过于限制性的。

见连接池描述。

因此，我所做的修正是将这些值分别增加到50和40。我仍然希望这个参数不受限制，并且随着使用而增长，但目前仍然坚持这些值。