高5分钟平均负荷的奇怪情况
高5分钟平均负荷的奇怪情况
提问于 2015-12-17 17:24:11
在这里寻求一些专家的建议。我是第一次在自己的服务器上使用sys管理员，我无法理解服务器中的瓶颈。
我收到了大量的5分钟负荷平均警报，从CSF到80之间。
因此，我继续在服务器上安装sqltuner，并让它运行3天。
结果并没有显示出任何异常，但我仍然得到高负荷平均每天5分钟
我正在寻找瓶颈(CPU、内存不足问题或I/O绑定负载所导致的负载)
如果有人能分享任何见解.
(我已经包括sqltuner的报告和下面的高负载电子邮件输出)
-------- Security Recommendations -------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- Performance Metrics -------------------------------------------------
[--] Up for: 120d 18h 27m 20s (227M q [21.795 qps], 51M conn, TX: 907B, RX: 26B)
[--] Reads / Writes: 38% / 62%
[--] Binary logging is disabled
[--] Total buffers: 15.4G global + 4.1M per thread (600 max threads)
[OK] Maximum reached memory usage: 16.2G (51.75% of installed RAM)
[OK] Maximum possible memory usage: 17.8G (56.91% of installed RAM)
[OK] Slow queries: 0% (11/227M)
[OK] Highest usage of available connections: 33% (199/600)
[OK] Aborted connections: 0.56% (284327/51183230)
[OK] Query cache efficiency: 83.0% (78M cached / 94M selects)
[!!] Query cache prunes per day: 10288
[OK] Sorts requiring temporary tables: 0% (392 temp sorts / 1M sorts)
[OK] Temporary tables created on disk: 4% (65K on disk / 1M total)
[OK] Thread cache hit rate: 99% (199 created / 51M connections)
[OK] Table cache hit rate: 22% (425 open / 1K opened)
[OK] Open file limit used: 0% (433/50K)
[OK] Table locks acquired immediately: 99% (41M immediate / 41M locks)
-------- MyISAM Metrics -----------------------------------------------------
[!!] Key buffer used: 20.2% (108M used / 536M cache)
[OK] Key buffer size / total MyISAM indexes: 512.0M/14.7M
[OK] Read Key buffer hit rate: 99.8% (51M cached / 121K reads)
[!!] Write Key buffer hit rate: 40.8% (4M cached / 2M writes)
-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 14.6G/140.9M
[!!] InnoDB buffer pool instances: 1
[!!] InnoDB Used buffer: 3.39% (32546 used/ 959999 total)
[OK] InnoDB Read buffer efficiency: 100.00% (5437258684 hits/ 5437259670 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 24069213 writes)
-------- AriaDB Metrics -----------------------------------------------------
[--] AriaDB is disabled.
-------- Replication Metrics -------------------------------------------------
[--] No replication slave(s) for this server.
[--] This is a standalone server..
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Increasing the query_cache size over 128M may reduce performance
Variables to adjust:
query_cache_size (> 128M) [see warning above]
innodb_buffer_pool_instances(=14)
----------------------
(The only change I've made is to reduce InnoDB size and add multiple pool instances)每日高负荷电子邮件：
Time: Sun Dec 6 05:43:53 2015 -0500
1 Min Load Avg: 80.26
5 Min Load Avg: 21.19
15 Min Load Avg: 7.46
Running/Total Processes: 221/875
ps.txt
O
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-1424 16243 0/149/2713129 W 1.28 14 0 0.0 2.34 65107.59 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
1-1424 17057 0/18/2701770 W 2.15 4 0 0.0 0.30 62402.50 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
2-1424 17064 0/24/2685073 W 2.11 13 0 0.0 0.32 62668.14 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
3-1424 15319 0/215/2657841 W 3.50 4 0 0.0 3.88 61950.21 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
4-1424 11567 0/204/2651294 W 7.10 7 0 0.0 3.00 63562.61 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
5-1424 16512 0/37/2640191 W 2.19 5 0 0.0 0.60 63637.48 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
6-1424 17735 0/8/2630311 W 0.62 19 0 0.0 0.06 65036.68 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
7-1424 16521 0/31/2613938 W 2.20 19 0 0.0 0.36 62385.07 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
8-1424 16081 0/33/2611913 W 2.46 5 0 0.0 0.42 60535.12 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
9-1424 14711 0/120/2603042 W 1.89 18 0 0.0 2.11 59868.26 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
10-1424 16838 0/21/2592501 W 1.77 15 0 0.0 0.24 62195.33 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
11-1424 16531 0/42/2584776 W 2.45 11 0 0.0 0.39 62253.11 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
12-1424 17065 0/20/2570161 W 1.29 12 0 0.0 0.18 60474.65 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
13-1424 17770 0/13/2564128 W 1.27 2 0 0.0 0.63 59748.24 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
14-1424 17771 0/14/2542936 W 1.30 2 0 0.0 0.17 60513.73 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
15-1424 15736 0/64/2536855 W 2.91 7 0 0.0 1.16 61453.61 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
16-1424 17077 0/19/2522131 W 2.76 15 0 0.0 0.35 59307.60 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
17-1424 14723 0/93/2521068 W 3.38 6 0 0.0 1.77 60437.40 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
18-1424 16279 0/62/2509938 W 1.81 15 0 0.0 1.07 61401.24 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
19-1424 15333 0/116/2498356 W 3.24 19 0 0.0 1.69 57911.45 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
20-1424 16297 1/35/2494463 W 0.98 53 62 16.1 0.47 59474.66 58.174.24.65 suspensionrevolution.com:80 GET /new/wp-content/themes/optimizePressTheme/lib/assets/defaul
21-1424 16298 0/40/2473943 W 3.83 3 0 0.0 0.54 57987.71 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
22-1424 18054 0/6/2469193 W 1.23 1 0 0.0 0.05 59122.65 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
23-1424 12894 0/162/2458774 W 5.90 17 0 0.0 2.42 56404.92 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
24-1424 18088 0/4/2452422 W 0.90 11 0 0.0 0.00 58405.08 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
25-1424 18089 0/6/2446570 W 1.22 1 0 0.0 0.03 57036.34 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
26-1424 17079 0/30/2439491 W 1.88 0 0 0.0 0.43 54697.67 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
27-1424 16101 0/64/2416961 W 1.53 18 0 0.0 1.69 57160.43 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
28-1424 18140 0/9/2403931 W 0.62 18 0 0.0 0.02 55901.03 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
29-982 1505 1/18/1548355 G 0.14 2914733 450294 2.8 0.29 34947.04 96.47.70.4 suspensionrevolution.com:80 POST /dap/dap-clickbank-6.0.php HTTP/1.1
30-1424 15338 0/100/2384316 W 2.20 7 0 0.0 1.24 53919.77 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
31-1424 16300 2/56/2379897 K 3.47 3 1365 2.4 1.01 55195.75 89.166.18.35 appcoiner.com:80 GET /favicon.ico HTTP/1.1
32-1424 15749 0/108/2369131 W 3.26 17 0 0.0 2.09 55452.02 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
33-1424 17100 0/17/2359616 W 1.70 11 0 0.0 0.16 52564.23 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
34-1424 16310 0/162/2356424 W 3.95 15 0 0.0 2.68 55800.32 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
35-1424 16543 0/63/2326471 W 1.29 4 0 0.0 0.75 55028.80 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
36-1424 17101 0/18/2331624 W 2.05 14 0 0.0 0.21 53656.66 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
37-1424 17102 0/20/2314444 W 1.51 19 0 0.0 0.29 55684.29 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
38-1424 19665 0/1/2295814 W 0.00 3 0 0.0 0.00 52187.61 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
39-984 8727 1/69/1464486 G 0.68 2906097 450284 2.8 0.83 33844.50 74.63.153.4 suspensionrevolution.com:80 POST /dap/dap-clickbank-6.0.php HTTP/1.1
40-1424 19720 0/1/2277467 W 0.00 2 0 0.0 0.00 55864.93 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
41-1424 18141 0/6/2270838 W 0.62 14 0 0.0 0.02 54059.95 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
42-983 18177 1/49/1440183 G 0.63 2910665 450302 2.8 0.57 31224.74 74.63.153.4 suspensionrevolution.com:80 POST /dap/dap-clickbank-6.0.php HTTP/1.1
43-1424 16104 2/57/2242969 W 2.62 5 0 8.8 0.83 56170.39 54.202.7.147 appcoiner.com:80 GET /start-2/?utm_expid=111102625-1.-ThtNpCTSByWcbkMGdBOow.1&ho
44-1424 16547 0/28/2247277 W 3.84 5 0 0.0 0.31 53028.08 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
45-1424 15797 0/80/2225028 W 3.24 5 0 0.0 1.63 51333.94 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
46-1424 19721 0/1/2205346 W 0.00 2 0 0.0 0.00 52025.79 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
47-1424 18142 0/11/2207016 W 0.94 1 0 0.0 0.07 51355.07 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
48-1424 17104 0/137/2172322 W 1.28 7 0 0.0 2.32 49665.11 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
49-1424 16314 0/63/2168481 W 4.14 5 0 0.0 1.16 49191.04 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
50-1424 19763 0/0/2141243 W 2.41 12 0 0.0 0.00 49538.97 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
51-1424 17106 0/20/2137681 W 1.24 7 0 0.0 0.29 49973.70 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
52-1424 16549 0/34/2125106 W 1.99 7 0 0.0 0.50 50442.63 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
53-1424 17107 0/20/2109740 W 2.18 2 0 0.0 0.31 48074.92 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
54-1424 18143 0/4/2087977 W 1.21 8 0 0.0 0.00 49243.64 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
55-1424 17114 0/30/2062106 W 0.34 17 0 0.0 0.46 48605.36 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
56-1424 17115 0/19/2064562 W 2.07 0 0 0.0 0.30 47600.46 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
57-1424 16569 0/41/2051051 W 3.50 8 0 0.0 0.54 47547.57 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
58-1424 17116 0/28/2023150 W 1.28 4 0 0.0 0.42 49170.14 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
59-1424 17117 0/30/2010767 W 1.41 1 0 0.0 0.51 47681.03 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
60-1424 17118 0/17/1999913 R 0.05 53 5 0.0 0.26 46914.84 65.30.135.196
61-1424 16572 0/35/1978028 W 2.73 16 0 0.0 0.45 45848.82 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
62-1424 16573 0/27/1957566 W 1.60 0 0 0.0 0.45 46768.01 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
63-1424 16574 0/43/1936669 W 2.37 3 0 0.0 0.54 43520.07 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
64-1424 16575 0/28/1922381 W 1.54 1 0 0.0 0.33 45007.49 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
65-1424 16576 0/32/1903916 W 2.15 13 0 0.0 0.73 45117.83 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
66-1424 17119 0/28/1878566 W 1.18 6 0 0.0 0.45 44448.25 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1
67-1424 16578 1/39/1869043 K 2.01 0 0 1.2 0.53 44966.25 114.79.47.51 suspensionrevolution.com:80 GET /favicon.ico HTTP/1.1
68-1424 16579 0/37/1841958 W 2.59 5 0 0.0 0.48 44262.26 72.5.231.11 appcoiner.com:80 GET /?hopc2s=nakt123 HTTP/1.1这是我的最高产出：
root@ns513521 [~]# top
top - 08:21:31 up 153 days, 3:51, 1 user, load average: 0.15, 0.27, 0.51
Tasks: 230 total, 2 running, 227 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.4%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32855908k total, 25102984k used, 7752924k free, 886004k buffers
Swap: 1569780k total, 63984k used, 1505796k free, 21254784k cached这是我的ioStat输出：
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 21.99 1476.57 549.50 19540299168 7271856568
sdb 19.53 982.85 549.49 13006647390 7271718616
sdc 19.46 978.26 549.49 12945853934 7271718616
md2 9.78 492.22 399.94 6513868322 5292584264
md1 20.05 27.80 140.15 367920858 1854711136回答 3
Server Fault用户
发布于 2015-12-17 23:30:18
如果这总是在凌晨4点到6点之间，那么罪魁祸首可能是一些日常的cron工作。特别是updatedb可以减缓磁盘访问速度，这会导致大量使用的DB跨过一个阈值，从而增加负载。
在一般情况下，top将显示足够多的数据来诊断任何高负载问题，而sqltuner和您的电子邮件(它似乎提供了http连接)只有在DB或http连接中出现问题时才会有用。但是，几个小时后，您的顶部输出就会被占用，问题已经消失了(您需要页面的其余部分)。
Server Fault用户
发布于 2015-12-17 23:38:16
尝试在后台运行atsar一两天。这将将性能数据捕获到一个文件中，供以后检查。在负载平均值较高的期间检查其数据。如果您不确定何时发生这种情况，则可以从sar中获得负载平均值。
如果您还启用了流程记帐，您可能能够识别导致此问题的程序。有几个程序可以最大限度地减少背景工作的影响。nice可以将程序限制为使用其他空闲的CPU周期。ionice可以重新安排IO，以减少对较高优先级进程的影响。但是，这两种方法都可以显著增加问题程序的运行时间。
Server Fault用户
发布于 2015-12-18 10:57:13
CSF还对文件做了一个md5sum，将它们与前一天的md5sum进行比较。我已经看到它实际上会在生产环境中引起问题(托管300名客户的公司共享服务器)。如果您的服务器有很多文件，这可能是罪魁祸首之一。如果它发生在一天中的某个特定时间，那么克隆乔布斯无疑是罪魁祸首。以下命令的输出是什么？
for u in `cat /etc/passwd | cut -d":" -f1`;do crontab -l -u $u;done 页面原文内容由Server Fault提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://serverfault.com/questions/743772
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号