vPC与HSRP奇怪的arp问题

Question

下面是使用vPC和HSRP (版本1)配置的场景。

我看到了非常奇怪的问题，我的主机配置为bond + vlan，我的连接模式是active-backup，我只为HSRP配置了VLAN 100，我看到我的主机无法平HSRP虚拟IP，但它可以在VLAN 100上平所有其他主机，这个问题最近才开始运行，几周前一切正常。

vPC配置

vpc domain 1
  peer-switch
  role priority 10
  peer-keepalive destination 10.5.0.117 source 10.5.0.116
  peer-gateway
  auto-recovery
  ip arp synchronize

HSRP配置

interface Vlan100
  description *** Public_1 VLAN ***
  no shutdown
  mtu 9216
  no autostate
  no ip redirects
  ip address 74.xx.xx.2/23
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  hsrp 1
    preempt
    priority 110
    ip 74.xx.xx.1

我在主机VLAN 10和vlan 100上配置了两个vlan，下面是我的主机接口输出。

bond0.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.146  netmask 255.255.0.0  broadcast 10.10.255.255
        inet6 fe80::6e3b:e5ff:feba:84e8  prefixlen 64  scopeid 0x20<link>
        ether 6c:3b:e5:ba:84:e8  txqueuelen 1000  (Ethernet)
        RX packets 18724100  bytes 861377042 (821.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1579  bytes 160270 (156.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0.100: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 74.xx.xx.179  netmask 255.255.254.0  broadcast 74.xx.xx.255
        inet6 fe80::6e3b:e5ff:feba:84e8  prefixlen 64  scopeid 0x20<link>
        ether 6c:3b:e5:ba:84:e8  txqueuelen 1000  (Ethernet)
        RX packets 338156  bytes 15584262 (14.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74  bytes 7230 (7.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

HOSR arp表

[root@host ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
74.xx.xx.171           ether   fc:15:b4:13:1e:40   C                     bond0.100
74.xx.xx.2             ether   fc:5b:39:f7:6d:4f   C                     bond0.100
74.xx.xx.170           ether   d8:9d:67:75:2a:98   C                     bond0.100
74.xx.xx.1             ether   00:00:0c:07:ac:01   C                     bond0.100
74.xx.xx.177           ether   6c:3b:e5:b0:f9:f0   C                     bond0.100

关于SW1交换机ARP和MAC表

sw1# show ip arp 74.xx.xx.179

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies       D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface
74.xx.xx.179  00:02:35  6c3b.e5ba.84e8  Vlan100

SW1 MAC表

sw1# show mac address-table address 6c3b.e5ba.84e8
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   10     6c3b.e5ba.84e8   dynamic  0         F      F    Po46

FHRP

sw1# show ip arp fhrp-non-active-learn

Flags: D - Static Adjacencies attached to down interface

IP ARP Table for context default
Address         Age       MAC Address     Interface

SW2

sw2# show ip arp 74.xx.xx.179

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies       D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface
74.xx.xx.179  00:05:04  6c3b.e5ba.84e8  Vlan100          *

MAC表

sw2# show mac address-table address 6c3b.e5ba.84e8
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+   10     6c3b.e5ba.84e8   dynamic  0         F      F    Po46

FHRP

sw2# show ip arp fhrp-non-active-learn

Flags: D - Static Adjacencies attached to down interface

IP ARP Table for context default
Address         Age       MAC Address     Interface
74.xx.xx.179  00:07:32  6c3b.e5ba.84e8  Vlan100

更新- 1

我们在show logging中看到了很多下面的日志

2018 Jul  5 14:15:05 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:15:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:17:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:19:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:19:08 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0

更新-2

Jul  5 14:31:13 10.5.0.116 : 2018 Jul  5 18:24:35 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:17 10.5.0.116 : 2018 Jul  5 18:24:39 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:25 10.5.0.116 : 2018 Jul  5 18:24:47 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:29 10.5.0.116 : 2018 Jul  5 18:24:51 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:33 10.5.0.116 : 2018 Jul  5 18:24:54 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 6c3b.e5b0.c998 in vlan 100 has moved between Po35 to Po36
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:33:35 10.5.0.116 : 2018 Jul  5 18:26:57 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:36:43 10.5.0.116 : 2018 Jul  5 18:30:05 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100

问题：

1. 为什么我不能从主机上点击HSRP VIP 74.xx.x.1 ip地址，即使在主机arp表上，我也能看到它的MAC地址？
2. 为什么在MAC表中我看不到VLAN 100 mac地址(但我可以看到vlan 10 MAC)
3. fhrp-non-active-learn是什么?为什么它只出现在SW2 (备用HSRP实例)上？

Answer 1

决议：

问题是主机在round-robin键合配置上运行，造成大量MAC抖动，导致MAC表的锁定，防止新的MAC设备出现。

我们重新配置了所有服务器，以使用active-backup修复此问题。

注:上面的图表不正确(我们认为它是主动备份的，但它是循环的)