为什么PMP (可怜的男人的剖析器)不能在nginx上工作？

Question

有一个非常有用的gdb“脚本”叫做可怜人的轮廓仪。它将此命令称为：

gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $pid

它适用于大多数linux进程，但不适用于nginx web服务器。

正常产出：

# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 5286
Using host libthread_db library "/lib/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 0xb7d996c0 (LWP 5286)]
[New Thread 0xb588ab90 (LWP 5292)]
[New Thread 0xb608bb90 (LWP 5291)]
[New Thread 0xb688cb90 (LWP 5290)]
[New Thread 0xb708db90 (LWP 5289)]
[New Thread 0xb788eb90 (LWP 5288)]
0xffffe410 in __kernel_vsyscall ()

Thread 6 (Thread 0xb788eb90 (LWP 5288)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2  0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3  0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4  0x08055537 in worker_libevent (arg=0x805f3a0) at thread.c:245
#5  0xb7ed2192 in start_thread () from /lib/libpthread.so.0
#6  0xb7e5d02e in clone () from /lib/libc.so.6

[cut]

Thread 1 (Thread 0xb7d996c0 (LWP 5286)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2  0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3  0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4  0x0804f439 in main (argc=1, argv=0xbfbaff14) at memcached.c:4681
#0  0xffffe410 in __kernel_vsyscall ()

Nginx产出：

# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()

但是，如果使用"bt“而不是”线程应用所有bt“，则效果很好：

# gdb -ex "set pagination 0" -ex "bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7c83778 in epoll_wait () from /lib/libc.so.6
#2  0x080664c0 in ngx_epoll_process_events (cycle=0x80d14b8, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3  0x0805f73b in ngx_process_events_and_timers (cycle=0x80d14b8) at src/event/ngx_event.c:245
#4  0x080652b3 in ngx_worker_process_cycle (cycle=0x80d14b8, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5  0x08063ba1 in ngx_spawn_process (cycle=0x80d14b8, proc=0x80651fb <ngx_worker_process_cycle>, data=0x0, name=0x80a881d "worker process", respawn=-3) at src/os/unix/ngx_process.c:196
#6  0x080648d2 in ngx_start_worker_processes (cycle=0x80d14b8, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:355
#7  0x0806581f in ngx_master_process_cycle (cycle=0xfffffffc) at src/os/unix/ngx_process_cycle.c:136
#8  0x0804d076 in main (argc=1, argv=0xbfd86544) at src/core/nginx.c:396

为什么会这样呢？据我所知，即使进程中只有一个执行线程，“线程应用所有bt”也必须工作。

更新：

手动gdb连接和发出“信息线程”命令。

# gdb -p 17461
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Attaching to process 17461
Reading symbols from /export/depo/apache/linux/nginx-0.8.34/sbin/nginx...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /export/depo/mysql/linux/mysql-proxy-0.7.2/lib/libpcre.so.0...done.
Loaded symbols for /opt/gnu/mysql-proxy/lib/libpcre.so.0
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
0xffffe410 in __kernel_vsyscall ()
(gdb) info threads
(gdb) thread 0
Thread ID 0 not known.
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7d0dd98 in __epoll_wait_nocancel () from /lib/libc.so.6
#2  0x08066e7d in ngx_epoll_process_events (cycle=0x816b708, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3  0x0805f4a2 in ngx_process_events_and_timers (cycle=0x816b708) at src/event/ngx_event.c:245
#4  0x08064d83 in ngx_worker_process_cycle (cycle=0x816b708, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5  0x08063661 in ngx_spawn_process (cycle=0x816b708, proc=0x8064ccb <ngx_worker_process_cycle>, data=0x0, name=0x80a42ed "worker process", respawn=-4) at src/os/unix/ngx_process.c:196
#6  0x080643a1 in ngx_start_worker_processes (cycle=0x816b708, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:355
#7  0x0806595c in ngx_master_process_cycle (cycle=0x816b708) at src/os/unix/ngx_process_cycle.c:249
#8  0x0804cf7f in main (argc=1, argv=0xbfdb1874) at src/core/nginx.c:396

Answer 1

thread apply all bt为每个线程执行(意外！) bt。当有任何bt导致错误的线程时，它也会停止(但这不应该发生)。

因此，如果您手动附加到nginx，然后执行info threads，然后继续重复thread N和bt，您可能会发现GDB无法生成回溯的线程。

确切地知道GDB为该线程打印了什么，这可能会很有趣。

您可能还需要重新尝试GDB的当前版本(7.2)。

Answer 2

nginx不是多线程的(请参阅OpenSSL锁上的这个线程)。这可能解释了为什么您没有像预期的那样看到大量的线程数据。

nginx有一个主进程，它使用forks (没有exec)来创建工作人员。请参阅第90行的ngx_process.c和ngx_spawn_process。

我不知道PMP (可怜人的分析器)如何处理它，但我相信您需要在set follow-fork-mode child in gdb。参见，例如，gdbserver跟踪子。