看门狗: BUG:软锁- CPU#6卡住了23s

Question

我试过在谷歌上找到的所有解决方案..。我不知道我的服务器为什么要崩溃..。

Aug  5 17:11:08  kernel: [ 2300.084576] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [VM Thread:4054]
Aug  5 17:11:08  kernel: [ 2300.084578] Modules linked in: veth nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter bridge stp llc rpcsec_gss_krb5 auth_rpcgss aufs nfsv4 nfs lockd grace fscache overlay isofs xt_nat xt_MASQUERADE xt_addrtype iptable_nat nf_nat xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ppdev kvm_intel kvm ipmi_si input_leds joydev ipmi_devintf ipmi_msghandler video parport_pc parport acpi_pad sch_fq_codel drm sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel crypto_simd cryptd glue_helper usbhid igb hid nvme dca ahci i2c_algo_bit nvme_core libahci
Aug  5 17:11:08  kernel: [ 2300.084616] CPU: 6 PID: 4054 Comm: VM Thread Not tainted 5.4.0-42-generic #46-Ubuntu
Aug  5 17:11:08  kernel: [ 2300.084616] Hardware name: Intel Corporation S1200SP/S1200SP, BIOS S1200SP.86B.03.01.0042.013020190050 01/30/2019
Aug  5 17:11:08  kernel: [ 2300.084620] RIP: 0010:_raw_spin_lock+0x10/0x30
Aug  5 17:11:08  kernel: [ 2300.084621] Code: ff 01 00 00 75 07 4c 89 e0 41 5c 5d c3 e8 f8 f9 62 ff 4c 89 e0 41 5c 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 01 c3 55 89 c6 48 89 e5 e8 c2 e1 62 ff 66 90 5d c3 66 66 2e 0f
Aug  5 17:11:08  kernel: [ 2300.084621] RSP: 0000:ffffa592c1bef760 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Aug  5 17:11:08  kernel: [ 2300.084622] RAX: 0000000000000000 RBX: 0000000000000100 RCX: ffff95314b79bc00
Aug  5 17:11:08  kernel: [ 2300.084622] RDX: 0000000000000001 RSI: 0000000000000588 RDI: ffff953145c1aeac
Aug  5 17:11:08  kernel: [ 2300.084623] RBP: ffffa592c1bef7b8 R08: ffff95314a5520f0 R09: 0000000000000000
Aug  5 17:11:08  kernel: [ 2300.084623] R10: 0000000000000000 R11: ffffffffffffffb8 R12: 0000000000000000
Aug  5 17:11:08  kernel: [ 2300.084623] R13: ffff953145c1ae00 R14: ffff95314b79bc00 R15: ffff953145c1aeac
Aug  5 17:11:08  kernel: [ 2300.084624] FS:  00007fa0e4151700(0000) GS:ffff953151580000(0000) knlGS:0000000000000000
Aug  5 17:11:08  kernel: [ 2300.084624] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  5 17:11:08  kernel: [ 2300.084625] CR2: 0000000594832008 CR3: 000000045bf00003 CR4: 00000000003606e0
Aug  5 17:11:08  kernel: [ 2300.084625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug  5 17:11:08  kernel: [ 2300.084625] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug  5 17:11:08  kernel: [ 2300.084626] Call Trace:
Aug  5 17:11:08  kernel: [ 2300.084628]  ? scan_swap_map_slots+0x3cd/0x510
Aug  5 17:11:08  kernel: [ 2300.084629]  get_swap_pages+0x207/0x380
Aug  5 17:11:08  kernel: [ 2300.084630]  ? rmap_walk_anon+0x16f/0x260
Aug  5 17:11:08  kernel: [ 2300.084632]  get_swap_page+0xe3/0x210
Aug  5 17:11:08  kernel: [ 2300.084633]  add_to_swap+0x1a/0x70
Aug  5 17:11:08  kernel: [ 2300.084634]  shrink_page_list+0x4b3/0xbb0
Aug  5 17:11:08  kernel: [ 2300.084648]  shrink_inactive_list+0x201/0x3e0
Aug  5 17:11:08  kernel: [ 2300.084649]  shrink_node_memcg+0x137/0x370
Aug  5 17:11:08  kernel: [ 2300.084650]  shrink_node+0xbd/0x400
Aug  5 17:11:08  kernel: [ 2300.084650]  do_try_to_free_pages+0xd7/0x3a0
Aug  5 17:11:08  kernel: [ 2300.084651]  try_to_free_mem_cgroup_pages+0xf4/0x210
Aug  5 17:11:08  kernel: [ 2300.084653]  try_charge+0x2eb/0x810
Aug  5 17:11:08  kernel: [ 2300.084654]  ? find_get_entry+0xaf/0x170
Aug  5 17:11:08  kernel: [ 2300.084655]  mem_cgroup_try_charge+0x71/0x190
Aug  5 17:11:08  kernel: [ 2300.084656]  ? pagecache_get_page+0x2d/0x300
Aug  5 17:11:08  kernel: [ 2300.084657]  mem_cgroup_try_charge_delay+0x22/0x50
Aug  5 17:11:08  kernel: [ 2300.084658]  do_swap_page+0x220/0x9f0
Aug  5 17:11:08  kernel: [ 2300.084659]  __handle_mm_fault+0x73b/0x7a0
Aug  5 17:11:08  kernel: [ 2300.084659]  handle_mm_fault+0xca/0x200
Aug  5 17:11:08  kernel: [ 2300.084661]  do_user_addr_fault+0x1f9/0x450
Aug  5 17:11:08  kernel: [ 2300.084662]  __do_page_fault+0x58/0x90
Aug  5 17:11:08  kernel: [ 2300.084663]  do_page_fault+0x2c/0xe0
Aug  5 17:11:08  kernel: [ 2300.084664]  page_fault+0x34/0x40
Aug  5 17:11:08  kernel: [ 2300.084665] RIP: 0033:0x7fa168646be3
Aug  5 17:11:08  kernel: [ 2300.084666] Code: 4c 89 6d b8 49 89 5d 00 49 c7 45 08 00 00 00 00 4c 3b 6d b0 0f 83 1d 01 00 00 4c 89 6d b0 49 89 dd 4d 39 fd 0f 83 bd 00 00 00 <49> 8b 45 00 4c 89 eb 83 e0 03 48 83 f8 03 0f 84 09 01 00 00 42 0f
Aug  5 17:11:08  kernel: [ 2300.084666] RSP: 002b:00007fa0e41501b0 EFLAGS: 00010283
Aug  5 17:11:08  kernel: [ 2300.084667] RAX: 00000005237c2908 RBX: 0000000000000004 RCX: 00007fa0e41504b0
Aug  5 17:11:08  kernel: [ 2300.084667] RDX: 0000000000000004 RSI: 0000000594831fe8 RDI: 00007fa160745850
Aug  5 17:11:08  kernel: [ 2300.084668] RBP: 00007fa0e4150230 R08: 00000005237c28e8 R09: 00007fa1607458f0
Aug  5 17:11:08  kernel: [ 2300.084668] R10: 00007fa168f52d99 R11: 000000014b7bf600 R12: 00007fa1609924d0
Aug  5 17:11:08  kernel: [ 2300.084668] R13: 0000000594832008 R14: 0000000000000240 R15: 0000000595000000

我换了我的硬件但是我的磁盘。

当我用“我的世界”启动一个码头容器(翼龙)时，有时它会因为上面的错误而冻结。我找不到相关的日志..。

uname -a：Linux X-X-X 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

免费-h：total used free shared buff/cache available Mem: 31Gi 594Mi 29Gi 4.0Mi 1.1Gi 30Gi Swap: 1.0Gi 0B 1.0Gi

sysctl vm.swappiness：vm.swappiness = 60

sudo内存：

  *-firmware
       description: BIOS
       vendor: Intel Corporation
       physical id: 6
       version: S1200SP.86B.03.01.0042.013020190050
       date: 01/30/2019
       size: 64KiB
       capacity: 16MiB
       capabilities: pci pnp upgrade shadowing cdboot bootselect edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer int10video acpi usb ls120boot zipboot biosbootspecification netboot uefi
  *-cache:0
       description: L1 cache
       physical id: 1a
       slot: L1 Cache
       size: 128KiB
       capacity: 128KiB
       capabilities: synchronous internal write-through instruction
       configuration: level=1
  *-cache:1
       description: L2 cache
       physical id: 1b
       slot: L2 Cache
       size: 1MiB
       capacity: 1MiB
       capabilities: synchronous internal write-through unified
       configuration: level=2
  *-cache:2
       description: L3 cache
       physical id: 1c
       slot: L3 Cache
       size: 8MiB
       capacity: 8MiB
       capabilities: synchronous internal write-back unified
       configuration: level=3
  *-cache
       description: L1 cache
       physical id: 19
       slot: L1 Cache
       size: 128KiB
       capacity: 128KiB
       capabilities: synchronous internal write-through data
       configuration: level=1
  *-memory
       description: System Memory
       physical id: 1e
       slot: System board or motherboard
       size: 32GiB
     *-bank:0
          description: [empty]
          vendor: Empty/NO DIMM
          physical id: 0
          slot: DIMM_A1
     *-bank:1
          description: DIMM DDR4 Synchronous 2400 MHz (0.4 ns)
          product: KHX2400C15/16G
          vendor: Kingston
          physical id: 1
          serial: A800F9241
          slot: DIMM_A2
          size: 16GiB
          width: 64 bits
          clock: 2400MHz (0.4ns)
     *-bank:2
          description: [empty]
          vendor: Empty/NO DIMM
          physical id: 2
          slot: DIMM_B1
     *-bank:3
          description: DIMM DDR4 Synchronous 2400 MHz (0.4 ns)
          product: KHX2400C15/16G
          vendor: Kingston
          physical id: 3
          serial: BE305496
          slot: DIMM_B2
          size: 16GiB
          width: 64 bits
          clock: 2400MHz (0.4ns)
  *-memory UNCLAIMED
       description: Memory controller
       product: 100 Series/C230 Series Chipset Family Power Management Controller
       vendor: Intel Corporation
       physical id: 1f.2
       bus info: pci@0000:00:1f.2
       version: 31
       width: 32 bits
       clock: 33MHz (30.3ns)
       capabilities: bus_master
       configuration: latency=0
       resources: memory:a2f10000-a2f13fff

grep -i交换/etc/fstab：

UUID="X-X-X-X-X" swap swap defaults 0 0
UUID="X-X-X-X-X" swap swap defaults 0 0
/swapfile swap swap defaults 0 0

有什么想法吗？

Answer 1

可能的交换/内存问题。

您的BIOS版本为S1200SP.86B.03.01.0042.013020190050，日期为01/30/2019。

有一个更新的BIOS，日期为2020年6月，可以下载这里。

注意:在更新BIOS之前有很好的备份。

Memtest

转到https://www.memtest86.com/并下载/运行他们的免费memtest来测试您的内存。在所有4/4测试中至少通过一次，以确认良好的记忆力。这可能需要很多小时才能完成。

更新#1:

就像我以前想的..。你有交换问题。

您有三个交换位置，如/etc/fstab中所示！

UUID="X-X-X-X-X" swap swap defaults 0 0
UUID="X-X-X-X-X" swap swap defaults 0 0
/swapfile swap swap defaults 0 0

sudo swapoff -a #关闭交换吗？

然后在/etc/fstab中注释掉上述三行。

完全禁用交换是永远不可能的。太小的掉期是不合适的。你两个都有问题。

让我们为您的系统创建一个合适的/swapfile。

注意:不正确地使用dd命令会导致数据丢失。建议复制/粘贴。

sudo swapoff -a           # turn off swap
sudo rm -i /swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096

sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 32G RAM and 4G swap

将这一行添加到/etc/fstab..。

/swapfile    none    swap    sw      0   0

然后对系统进行reboot验证操作。

如果一切正常，您可以使用gparted删除两个磁盘分区，其中的UUID显示在/etc/fstab中注释掉的行中。这里要小心，并确保您有正确的分区要删除。然后删除/etc/fstab中的这三行注释行。

Answer 2

虽然这个问题似乎得到了回答，但是对于那些发现自己在这里有相同CPU错误的人(除了heynnemas的答案)，请检查您连接过的任何显卡的PCI电缆连接。

我有同样的错误和问题停止后，断开图形卡，我后来意识到有一个错误(和烧焦) 6-Pin连接。更换电缆返回的系统功能正常。

我还建议检查CPU /内存的时间不是太疯狂，并且CPU冷却器连接正确(紧密)。

Answer 3

我在本地运行的VM场中的VM上出现了这个错误，它的磁盘已经满了。虚拟机监控程序无法为“瘦”磁盘分区分配更多空间(这些分区具有按需分配的物理空间，而且场被超额订阅)。请注意，系统管理程序需要一定的开销才能运行(可能是10%)，并将保留该空间。

结果发现其中一台物理机器出了问题，没有报告释放了磁盘空间，这导致VM场陷入混乱，认为磁盘已满。当那台机器重新启动时，问题就消失了。我们正在进行操作系统和管理程序更新--希望这能防止将来的问题。