Help with ARP issue

Hi,


I have a 3750x switch that have very high CPU. The cause of this seems to be the ARP Input with is running at around 40%-60% and the cpu is running at around 99%. What can I do to find out that is causing the ARP Input to go so high?

CPU utilization for five seconds: 99%/12%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  12   111438973    18587995       5995 44.47% 43.88% 43.96%   0 ARP Input
 174    59541847     5198737      11453 22.39% 23.47% 23.62%   0 Hulc LED Process
 221     7253246     6147816       1179  4.95%  4.25%  4.10%   0 IP Input
  86     5459437     1100349       4961  1.59%  1.47%  1.54%   0 RedEarth Tx Mana
  85     3448684     1453278       2373  1.27%  1.04%  1.07%   0 RedEarth I2C dri

any ideas would be much appreciated

«1

Comments

  • You might have something like the following in your configuration :

     

    Ip route 0.0.0.0 0.0.0.0 f1/0

     

    In this case, the router will arp for every destination of which the next hop is unknown. 

  • Thank you for you reply, There are no configured static routes as it is using OSPF for routeing.

  • Hi,


    I have a 3750x switch that have very high CPU. The cause of this seems to be the ARP Input with is running at around 40%-60% and the cpu is running at around 99%. What can I do to find out that is causing the ARP Input to go so high?

    any ideas would be much appreciated

    Firstly, check if you have loop scenario.

    Is it operating in layer 2 or layer 3 mode?

     

  • Thank you for you reply, There are no configured static routes as it is using OSPF for routeing.


     

    Interesting. Can you paste "sh arp" and "sh ip route"?

  • The switch is oprating at both layer 2 and layer 3. There are around 20 switches connected back to this 3750x stack. I have noticed a number of mac flaps as well.

     

    I have included arp for vlan 1 as this is the problem vlan.

    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.15.224          -   6c20.5660.a940  ARPA   Vlan1
    Internet  192.168.0.141           0   0023.54fa.598d  ARPA   Vlan1
    Internet  192.168.0.208           0   0026.180a.733f  ARPA   Vlan1
    Internet  192.168.1.189           0   0026.185c.a63a  ARPA   Vlan1
    Internet  192.168.1.138           0   7071.bc06.2682  ARPA   Vlan1
    Internet  192.168.0.143           0   0023.54fa.5997  ARPA   Vlan1
    Internet  192.168.0.151           0   7071.bc06.2630  ARPA   Vlan1
    Internet  192.168.1.34            0   90e6.ba17.b2a0  ARPA   Vlan1
    Internet  192.168.1.177           0   00e0.4c68.0050  ARPA   Vlan1
    Internet  192.168.15.26           0   f8d1.1100.133f  ARPA   Vlan1
    Internet  192.168.2.130           0   001c.c06c.d587  ARPA   Vlan1
    Internet  192.168.1.204           0   001c.c06c.d8b6  ARPA   Vlan1
    Internet  192.168.0.83            0   14da.e968.d143  ARPA   Vlan1
    Internet  192.168.15.10           0   60a4.4c64.8b1d  ARPA   Vlan1
    Internet  192.168.14.152          0   0014.fd11.843e  ARPA   Vlan1
    Internet  192.168.15.11           0   60a4.4c64.8c21  ARPA   Vlan1
    Internet  192.168.0.9             0   902b.3440.5ad5  ARPA   Vlan1
    Internet  192.168.0.77            0   0026.180a.7039  ARPA   Vlan1
    Internet  192.168.1.41            0   7071.bc07.337d  ARPA   Vlan1
    Internet  192.168.3.31            2   6805.ca05.7418  ARPA   Vlan1
    Internet  192.168.14.31           1   0011.4335.39c4  ARPA   Vlan1
    Internet  192.168.0.200           0   0019.d1e4.5717  ARPA   Vlan1
    Internet  192.168.15.25           0   3cd9.2b5d.2cc3  ARPA   Vlan1
    Internet  192.168.15.30           0   f8d1.1102.ef7c  ARPA   Vlan1
    Internet  192.168.15.101          6   0090.0b18.f872  ARPA   Vlan1
    Internet  192.168.15.100        169   0090.0b18.f870  ARPA   Vlan1
    Internet  192.168.14.3            0   0001.8021.1cba  ARPA   Vlan1
    Internet  192.168.15.225          1   6c20.566f.bcc0  ARPA   Vlan1
    Internet  192.168.14.26           1   0003.4711.5043  ARPA   Vlan1
    Internet  192.168.1.5             0   90e6.ba17.b8f8  ARPA   Vlan1
    Internet  192.168.14.30           0   0024.21a8.0960  ARPA   Vlan1
    Internet  192.168.14.151          0   902b.3440.0a2d  ARPA   Vlan1
    Internet  192.168.1.205           0   001c.c06c.d576  ARPA   Vlan1
    Internet  192.168.1.75            0   001c.c06c.d858  ARPA   Vlan1
    Internet  192.168.3.17            0   a41f.7286.4779  ARPA   Vlan1
    Internet  192.168.0.7             0   f46d.0494.5349  ARPA   Vlan1
    Internet  192.168.1.223           0   7071.bc07.335c  ARPA   Vlan1
    Internet  192.168.0.173           0   7071.bc50.0b4d  ARPA   Vlan1
    Internet  192.168.0.48            0   0026.1823.9e4d  ARPA   Vlan1
    Internet  192.168.0.97            0   0026.185c.a8ea  ARPA   Vlan1
    Internet  192.168.2.99            0   001c.c06c.d5f2  ARPA   Vlan1
    Internet  192.168.2.95            0   001c.c06c.d878  ARPA   Vlan1
    Internet  192.168.2.97            4   001c.c06c.d782  ARPA   Vlan1
    Internet  192.168.1.116           0   7071.bc06.24b9  ARPA   Vlan1
    Internet  192.168.0.233           0   0026.180a.72c1  ARPA   Vlan1
    Internet  192.168.2.58            0   001c.c06c.d802  ARPA   Vlan1
    Internet  192.168.0.220           0   0026.180a.720c  ARPA   Vlan1
    Internet  192.168.2.87            0   001c.c06c.d6f5  ARPA   Vlan1
    Internet  192.168.1.187           0   7071.bc50.0e45  ARPA   Vlan1
    Internet  192.168.0.106           0   0026.185c.a60f  ARPA   Vlan1
    Internet  192.168.1.73            0   0026.185c.a8ce  ARPA   Vlan1
    Internet  192.168.1.79            0   0026.180a.7069  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.0.98            0   7071.bc07.36a7  ARPA   Vlan1
    Internet  192.168.1.11            0   001c.c06c.e296  ARPA   Vlan1
    Internet  192.168.0.81            0   0026.1823.9b3a  ARPA   Vlan1
    Internet  192.168.2.66            0   001c.c06c.da67  ARPA   Vlan1
    Internet  192.168.0.203           0   001c.c06c.d8fb  ARPA   Vlan1
    Internet  192.168.0.210           0   0026.180a.734d  ARPA   Vlan1
    Internet  192.168.1.181           0   001c.c06c.dc1b  ARPA   Vlan1
    Internet  192.168.1.45            0   001c.c06c.d808  ARPA   Vlan1
    Internet  192.168.0.157           0   0024.8c63.85ba  ARPA   Vlan1
    Internet  192.168.1.167           0   001c.c06c.d6fd  ARPA   Vlan1
    Internet  192.168.1.126           0   001c.c06c.e284  ARPA   Vlan1
    Internet  192.168.0.252           0   001c.c06c.d8d5  ARPA   Vlan1
    Internet  192.168.1.194           0   7071.bc51.c7a9  ARPA   Vlan1
    Internet  192.168.0.105           0   0026.180a.7359  ARPA   Vlan1
    Internet  192.168.0.218           0   0026.180a.7205  ARPA   Vlan1
    Internet  192.168.1.3             0   001f.c6f7.dfa5  ARPA   Vlan1
    Internet  192.168.1.63            0   001c.c06c.d78c  ARPA   Vlan1
    Internet  192.168.1.8             0   001f.c6f7.dec2  ARPA   Vlan1
    Internet  192.168.0.19            0   001c.c06c.d5d5  ARPA   Vlan1
    Internet  192.168.1.243           0   7071.bc07.36a9  ARPA   Vlan1
    Internet  192.168.0.225           0   001c.c06c.d90f  ARPA   Vlan1
    Internet  192.168.0.180           0   001c.c06c.d831  ARPA   Vlan1
    Internet  192.168.1.219           0   7071.bc07.3324  ARPA   Vlan1
    Internet  192.168.3.4           169   0026.180a.7188  ARPA   Vlan1
    Internet  192.168.0.206         169   7071.bc51.c89c  ARPA   Vlan1
    Internet  192.168.2.109           0   001c.c06c.d68b  ARPA   Vlan1
    Internet  192.168.0.8             0   001c.c06c.d7b0  ARPA   Vlan1
    Internet  192.168.1.152         169   001c.c06c.d7fc  ARPA   Vlan1
    Internet  192.168.2.71            0   7071.bc50.0c3d  ARPA   Vlan1
    Internet  192.168.1.13            0   0026.1823.99e5  ARPA   Vlan1
    Internet  192.168.1.144           0   001c.c06c.d5d9  ARPA   Vlan1
    Internet  192.168.1.250           0   7071.bc06.2529  ARPA   Vlan1
    Internet  192.168.1.59            0   0019.d1e4.569c  ARPA   Vlan1
    Internet  192.168.2.117          86   60a4.4c64.1154  ARPA   Vlan1
    Internet  192.168.0.246           0   7071.bc37.9c42  ARPA   Vlan1
    Internet  192.168.0.109           0   0026.1823.9a1c  ARPA   Vlan1
    Internet  192.168.1.23            0   0026.1823.99fd  ARPA   Vlan1
    Internet  192.168.0.154           0   7071.bc50.0e4c  ARPA   Vlan1
    Internet  192.168.0.175           0   7071.bc37.9d97  ARPA   Vlan1
    Internet  192.168.1.67            0   7071.bc37.9da5  ARPA   Vlan1
    Internet  192.168.0.49            0   0026.180a.7036  ARPA   Vlan1
    Internet  192.168.0.130           0   7071.bc50.0a89  ARPA   Vlan1
    Internet  192.168.1.52            0   90e2.ba40.1761  ARPA   Vlan1
    Internet  192.168.0.229           0   0023.54fa.59c5  ARPA   Vlan1
    Internet  192.168.1.36            0   0023.54fa.59cf  ARPA   Vlan1
    Internet  192.168.0.211           0   0026.1823.9a1f  ARPA   Vlan1
    Internet  192.168.2.0             0   001c.c065.dd9d  ARPA   Vlan1
    Internet  192.168.3.5             0   0026.1823.9a36  ARPA   Vlan1
    Internet  192.168.0.39            0   0026.180a.7373  ARPA   Vlan1
    Internet  192.168.0.93            0   0026.185c.a8a1  ARPA   Vlan1
    Internet  192.168.0.144           0   7071.bc06.24a5  ARPA   Vlan1
    Internet  192.168.3.10            0   0026.1823.9cdc  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.0.31            0   14da.e968.d02a  ARPA   Vlan1
    Internet  192.168.2.140           0   0023.54fa.59a7  ARPA   Vlan1
    Internet  192.168.1.62            0   001c.c06c.dfa1  ARPA   Vlan1
    Internet  192.168.0.114           0   0026.180a.7328  ARPA   Vlan1
    Internet  192.168.0.30            0   7071.bc06.2631  ARPA   Vlan1
    Internet  192.168.2.84            0   7071.bc07.36ad  ARPA   Vlan1
    Internet  192.168.0.248           0   0026.1823.9b5d  ARPA   Vlan1
    Internet  192.168.0.107           0   0026.1823.9ce2  ARPA   Vlan1
    Internet  192.168.0.249           0   0026.185c.a888  ARPA   Vlan1
    Internet  192.168.2.155           0   7071.bc30.b168  ARPA   Vlan1
    Internet  192.168.0.67            0   14da.e968.cb95  ARPA   Vlan1
    Internet  192.168.0.13            0   0026.180a.72fd  ARPA   Vlan1
    Internet  192.168.1.111           0   001c.c06c.d7c6  ARPA   Vlan1
    Internet  192.168.2.138           0   0026.1823.99e4  ARPA   Vlan1
    Internet  192.168.0.240           0   0026.1823.9aad  ARPA   Vlan1
    Internet  192.168.0.124           0   0026.1823.9a0e  ARPA   Vlan1
    Internet  192.168.1.22            0   0026.184e.137f  ARPA   Vlan1
    Internet  192.168.0.75            0   7071.bc37.9ded  ARPA   Vlan1
    Internet  192.168.0.62            0   0026.180a.7215  ARPA   Vlan1
    Internet  192.168.2.96            0   001c.c06c.d6d5  ARPA   Vlan1
    Internet  192.168.0.22            0   0026.180a.7210  ARPA   Vlan1
    Internet  192.168.2.175           0   7071.bc50.1465  ARPA   Vlan1
    Internet  192.168.0.137           0   0026.180a.73bb  ARPA   Vlan1
    Internet  192.168.0.145           0   7071.bc30.afe7  ARPA   Vlan1
    Internet  192.168.2.47            0   7071.bc50.0b73  ARPA   Vlan1
    Internet  192.168.1.218           0   7071.bc07.32f9  ARPA   Vlan1
    Internet  192.168.1.77          169   7071.bc07.36f5  ARPA   Vlan1
    Internet  192.168.0.239           0   0026.180a.7326  ARPA   Vlan1
    Internet  192.168.1.40            0   0026.185c.aaf8  ARPA   Vlan1
    Internet  192.168.0.149           0   7071.bc06.24a7  ARPA   Vlan1
    Internet  192.168.0.111           0   0023.54fa.5a3a  ARPA   Vlan1
    Internet  192.168.0.96            0   001f.c6f7.ded1  ARPA   Vlan1
    Internet  192.168.0.243           0   0026.180a.7357  ARPA   Vlan1
    Internet  192.168.1.82            0   7071.bc50.0ee6  ARPA   Vlan1
    Internet  192.168.0.244         142   7071.bc37.9cc1  ARPA   Vlan1
    Internet  192.168.0.190           0   001c.c06c.d81e  ARPA   Vlan1
    Internet  192.168.2.238           0   14da.e968.c925  ARPA   Vlan1
    Internet  192.168.0.82            0   14da.e968.c8cf  ARPA   Vlan1
    Internet  192.168.1.246           0   0026.1823.99ec  ARPA   Vlan1
    Internet  192.168.1.161           0   7071.bc06.24f0  ARPA   Vlan1
    Internet  192.168.0.214           0   0026.1823.9a46  ARPA   Vlan1
    Internet  192.168.0.116           0   0024.8c8f.94a1  ARPA   Vlan1
    Internet  192.168.0.162           0   7071.bc07.32ff  ARPA   Vlan1
    Internet  192.168.0.202           0   7071.bc07.3317  ARPA   Vlan1
    Internet  192.168.3.9             0   0026.180a.7045  ARPA   Vlan1
    Internet  192.168.0.122           0   0026.180a.7074  ARPA   Vlan1
    Internet  192.168.0.131           0   7071.bc07.3303  ARPA   Vlan1
    Internet  192.168.0.132           0   0026.1823.9c86  ARPA   Vlan1
    Internet  192.168.2.42            0   0026.1823.9a03  ARPA   Vlan1
    Internet  192.168.2.193           0   0023.54fa.59fe  ARPA   Vlan1
    Internet  192.168.0.99            0   0026.1823.9c8c  ARPA   Vlan1
    Internet  192.168.1.38            0   7071.bc50.0d74  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.1.91          169   0023.54fa.59a0  ARPA   Vlan1
    Internet  192.168.1.227           0   7071.bc06.24d5  ARPA   Vlan1
    Internet  192.168.0.85            0   14da.e968.d025  ARPA   Vlan1
    Internet  192.168.0.68            0   7071.bc06.2543  ARPA   Vlan1
    Internet  192.168.1.64            0   0026.1823.9ac8  ARPA   Vlan1
    Internet  192.168.0.170           0   7071.bc07.32f0  ARPA   Vlan1
    Internet  192.168.1.83            0   14da.e968.c829  ARPA   Vlan1
    Internet  192.168.1.222           0   0023.54fa.5a06  ARPA   Vlan1
    Internet  192.168.2.104           0   001c.c06c.d766  ARPA   Vlan1
    Internet  192.168.2.1             0   001c.c06c.d723  ARPA   Vlan1
    Internet  192.168.0.73            0   14da.e968.d026  ARPA   Vlan1
    Internet  192.168.0.10            0   14da.e968.c7f8  ARPA   Vlan1
    Internet  192.168.1.15            0   0026.180a.7294  ARPA   Vlan1
    Internet  192.168.1.226           0   7071.bc07.331b  ARPA   Vlan1
    Internet  192.168.0.241           0   0026.1823.9ce5  ARPA   Vlan1
    Internet  192.168.0.167           0   1c6f.658f.470d  ARPA   Vlan1
    Internet  192.168.1.72            0   7071.bc50.1449  ARPA   Vlan1
    Internet  192.168.0.153           0   7071.bc07.3353  ARPA   Vlan1
    Internet  192.168.0.43            0   0023.54fa.59e2  ARPA   Vlan1
    Internet  192.168.0.61            0   14da.e968.cc1a  ARPA   Vlan1
    Internet  192.168.2.6           169   001c.c06c.d6df  ARPA   Vlan1
    Internet  192.168.1.233           0   7071.bc06.2528  ARPA   Vlan1
    Internet  192.168.0.15            0   14da.e968.cb62  ARPA   Vlan1
    Internet  192.168.2.85            0   0023.54fa.52ef  ARPA   Vlan1
    Internet  192.168.0.69            0   0019.d189.315a  ARPA   Vlan1
    Internet  192.168.0.205           0   001c.c06c.d620  ARPA   Vlan1
    Internet  192.168.2.53            0   001c.c06c.d695  ARPA   Vlan1
    Internet  192.168.2.8             0   14da.e968.d178  ARPA   Vlan1
    Internet  192.168.2.173           0   0026.1823.9a93  ARPA   Vlan1
    Internet  192.168.0.52            0   14da.e968.cfa4  ARPA   Vlan1
    Internet  192.168.2.14          169   14da.e968.d029  ARPA   Vlan1
    Internet  192.168.3.2             0   0026.1823.99e9  ARPA   Vlan1
    Internet  192.168.1.119           0   14da.e968.c8dc  ARPA   Vlan1
    Internet  192.168.2.240           0   0026.180a.7065  ARPA   Vlan1
    Internet  192.168.0.238           0   7071.bc51.c766  ARPA   Vlan1
    Internet  192.168.2.18          169   14da.e968.c826  ARPA   Vlan1
    Internet  192.168.0.53            0   0026.180a.72fb  ARPA   Vlan1
    Internet  192.168.1.117           0   14da.e968.c82e  ARPA   Vlan1
    Internet  192.168.1.105           0   001c.c06c.d8a6  ARPA   Vlan1
    Internet  192.168.2.24            0   001c.c06c.d794  ARPA   Vlan1
    Internet  192.168.2.48            0   0026.180a.7063  ARPA   Vlan1
    Internet  192.168.0.34            0   0026.180a.7355  ARPA   Vlan1
    Internet  192.168.0.138           0   0024.8c58.9abc  ARPA   Vlan1
    Internet  192.168.2.60            0   0026.1823.9ab1  ARPA   Vlan1
    Internet  192.168.2.172           0   0026.1823.9a98  ARPA   Vlan1
    Internet  192.168.1.113           0   001c.c06c.d97b  ARPA   Vlan1
    Internet  192.168.1.228         169   0026.1823.9ad5  ARPA   Vlan1
    Internet  192.168.0.91            0   0026.1823.9ab0  ARPA   Vlan1
    Internet  192.168.2.123           0   7071.bc06.2641  ARPA   Vlan1
    Internet  192.168.2.197           0   001c.c06c.d8f3  ARPA   Vlan1
    Internet  192.168.0.212           0   0026.1823.9acf  ARPA   Vlan1
    Internet  192.168.0.102           0   0026.185c.aaf0  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.2.242           0   14da.e968.cc5b  ARPA   Vlan1
    Internet  192.168.2.112           0   001c.c06c.d6e3  ARPA   Vlan1
    Internet  192.168.1.212         169   0024.8c63.83b3  ARPA   Vlan1
    Internet  192.168.0.232           0   001c.c06c.e223  ARPA   Vlan1
    Internet  192.168.0.58            0   14da.e968.cb8f  ARPA   Vlan1
    Internet  192.168.2.93            0   001c.c06c.dc23  ARPA   Vlan1
    Internet  192.168.1.90            0   7071.bc50.12f4  ARPA   Vlan1
    Internet  192.168.0.193           0   14da.e968.ca22  ARPA   Vlan1
    Internet  192.168.1.31            0   7071.bc07.349f  ARPA   Vlan1
    Internet  192.168.0.46            0   0026.180a.717a  ARPA   Vlan1
    Internet  192.168.0.195          31   0026.180a.7293  ARPA   Vlan1
    Internet  192.168.2.102           0   001c.c06c.d67e  ARPA   Vlan1
    Internet  192.168.0.24            0   001c.c06c.d737  ARPA   Vlan1
    Internet  192.168.2.5           169   14da.e968.d02b  ARPA   Vlan1
    Internet  192.168.2.125           0   7071.bc51.c8b2  ARPA   Vlan1
    Internet  192.168.0.27            0   14da.e968.d185  ARPA   Vlan1
    Internet  192.168.0.72            0   14da.e968.c834  ARPA   Vlan1
    Internet  192.168.0.188           0   001c.c06c.d47c  ARPA   Vlan1
    Internet  192.168.2.3             0   14da.e968.c7c9  ARPA   Vlan1
    Internet  192.168.0.254          42   0023.54fa.57f0  ARPA   Vlan1
    Internet  192.168.0.51          169   0026.1823.9cd4  ARPA   Vlan1
    Internet  192.168.2.29            0   001f.c6f7.decb  ARPA   Vlan1
    Internet  192.168.0.32            0   14da.e968.cdac  ARPA   Vlan1
    Internet  192.168.1.18            0   001c.c06c.d56d  ARPA   Vlan1
    Internet  192.168.1.21            0   001c.c06c.e225  ARPA   Vlan1
    Internet  192.168.1.74            0   0026.180a.7059  ARPA   Vlan1
    Internet  192.168.1.98            0   7071.bc06.266b  ARPA   Vlan1
    Internet  192.168.2.171           0   0026.1823.9a2f  ARPA   Vlan1
    Internet  192.168.0.64            0   0026.1823.9a0d  ARPA   Vlan1
    Internet  192.168.0.36            0   001c.c06c.d80a  ARPA   Vlan1
    Internet  192.168.0.215           0   0026.180a.72a1  ARPA   Vlan1
    Internet  192.168.0.217           0   0026.185c.a5ef  ARPA   Vlan1
    Internet  192.168.3.6             0   0026.1823.9b31  ARPA   Vlan1
    Internet  192.168.2.243         169   0026.1823.9cd3  ARPA   Vlan1
    Internet  192.168.0.95            0   0023.54fa.5979  ARPA   Vlan1
    Internet  192.168.0.139           0   14da.e968.d023  ARPA   Vlan1
    Internet  192.168.0.134           0   0026.1823.9a8e  ARPA   Vlan1
    Internet  192.168.1.25            0   0026.1823.9acc  ARPA   Vlan1
    Internet  192.168.0.223           0   0026.185c.a620  ARPA   Vlan1
    Internet  192.168.2.52            0   0026.1823.9a3e  ARPA   Vlan1
    Internet  192.168.0.209           0   0026.1823.9a8a  ARPA   Vlan1
    Internet  192.168.2.91            0   001c.c06c.dc33  ARPA   Vlan1
    Internet  192.168.0.89            0   90e6.ba17.bb11  ARPA   Vlan1
    Internet  192.168.0.2             0   001c.c06c.e04a  ARPA   Vlan1
    Internet  192.168.0.172           0   001c.c06c.d79a  ARPA   Vlan1
    Internet  192.168.2.126           0   7071.bc37.a620  ARPA   Vlan1
    Internet  192.168.0.71            0   0026.1823.9a9c  ARPA   Vlan1
    Internet  192.168.2.170           0   0026.185c.a773  ARPA   Vlan1
    Internet  192.168.1.207           0   001c.c06c.d879  ARPA   Vlan1
    Internet  192.168.0.169           0   14da.e968.c8e1  ARPA   Vlan1
    Internet  192.168.2.116         169   001c.c06c.d64d  ARPA   Vlan1
    Internet  192.168.1.249           0   7071.bc07.3364  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.2.244           0   0026.1823.9cc1  ARPA   Vlan1
    Internet  192.168.0.186           0   0026.180a.7307  ARPA   Vlan1
    Internet  192.168.0.135           0   0026.180a.7217  ARPA   Vlan1
    Internet  192.168.1.170           0   7071.bc07.32fe  ARPA   Vlan1
    Internet  192.168.2.98            0   001c.c06c.d5e7  ARPA   Vlan1
    Internet  192.168.0.140           0   0026.180a.720b  ARPA   Vlan1
    Internet  192.168.0.216           0   0026.1823.9bff  ARPA   Vlan1
    Internet  192.168.1.248           0   0026.1823.99e8  ARPA   Vlan1
    Internet  192.168.2.27            0   90e6.ba17.b66c  ARPA   Vlan1
    Internet  192.168.2.10          170   14da.e968.c7e1  ARPA   Vlan1
    Internet  192.168.0.191           0   0026.180a.72ad  ARPA   Vlan1
    Internet  192.168.1.171           0   7071.bc50.0c98  ARPA   Vlan1
    Internet  192.168.1.237           0   7071.bc07.3347  ARPA   Vlan1
    Internet  192.168.2.124           0   7071.bc50.0f15  ARPA   Vlan1
    Internet  192.168.1.143           0   7071.bc50.1364  ARPA   Vlan1
    Internet  192.168.0.28            0   14da.e968.ca47  ARPA   Vlan1
    Internet  192.168.1.19            0   0026.1823.9c04  ARPA   Vlan1
    Internet  192.168.2.55            0   001c.c06c.dc04  ARPA   Vlan1
    Internet  192.168.1.236           0   0026.1823.9a22  ARPA   Vlan1
    Internet  192.168.0.110           0   0026.1823.9aac  ARPA   Vlan1
    Internet  192.168.0.65            0   0023.54fa.59ec  ARPA   Vlan1
    Internet  192.168.1.27            0   0023.54fa.57cf  ARPA   Vlan1
    Internet  192.168.2.198           0   001c.c06c.d678  ARPA   Vlan1
    Internet  192.168.1.12            0   0023.54fa.5441  ARPA   Vlan1
    Internet  192.168.2.15            0   14da.e968.cb5a  ARPA   Vlan1
    Internet  192.168.1.118           0   14da.e968.cbcd  ARPA   Vlan1
    Internet  192.168.1.255           0   001c.c06c.d6d9  ARPA   Vlan1
    Internet  192.168.0.86            0   7071.bc07.3318  ARPA   Vlan1
    Internet  192.168.0.70            0   7071.bc07.3358  ARPA   Vlan1
    Internet  192.168.1.220           0   7071.bc06.267f  ARPA   Vlan1
    Internet  192.168.3.3             0   0026.1823.9bca  ARPA   Vlan1
    Internet  192.168.0.127           0   0026.1823.9a96  ARPA   Vlan1
    Internet  192.168.2.44            0   0026.180a.7020  ARPA   Vlan1
    Internet  192.168.0.194           0   14da.e968.d156  ARPA   Vlan1
    Internet  192.168.1.230           0   7071.bc06.2688  ARPA   Vlan1
    Internet  192.168.1.157           0   0026.1823.9a10  ARPA   Vlan1
    Internet  192.168.0.74          169   001c.c06c.dffa  ARPA   Vlan1
    Internet  192.168.1.58            0   0019.d1e4.5691  ARPA   Vlan1
    Internet  192.168.1.20            0   0026.184e.137b  ARPA   Vlan1
    Internet  192.168.0.245           0   7071.bc07.3357  ARPA   Vlan1
    Internet  192.168.1.229           0   7071.bc07.36f1  ARPA   Vlan1
    Internet  192.168.0.63            0   0026.180a.7295  ARPA   Vlan1
    Internet  192.168.1.43            0   7071.bc06.2501  ARPA   Vlan1
    Internet  192.168.0.84            0   14da.e968.c801  ARPA   Vlan1
    Internet  192.168.0.199           0   0026.1823.9c05  ARPA   Vlan1
    Internet  192.168.0.101           0   14da.e968.cc72  ARPA   Vlan1
    Internet  192.168.2.54            0   001c.c06c.d870  ARPA   Vlan1
    Internet  192.168.2.94            0   001c.c06c.d814  ARPA   Vlan1
    Internet  192.168.0.88            0   7071.bc50.1478  ARPA   Vlan1
    Internet  192.168.0.204           0   14da.e968.cb61  ARPA   Vlan1
    Internet  192.168.2.4             0   14da.e968.c755  ARPA   Vlan1
    Internet  192.168.1.26          169   0023.54fa.5522  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.1.192           0   001c.c06c.d694  ARPA   Vlan1
    Internet  192.168.1.32            0   7071.bc50.0e60  ARPA   Vlan1
    Internet  192.168.1.2             0   0023.54fa.5711  ARPA   Vlan1
    Internet  192.168.0.242           0   0026.180a.7146  ARPA   Vlan1
    Internet  192.168.2.127           0   7071.bc50.0eb5  ARPA   Vlan1
    Internet  192.168.0.165           0   0026.180a.7047  ARPA   Vlan1
    Internet  192.168.2.129           0   0026.1823.9a94  ARPA   Vlan1
    Internet  192.168.2.90            0   001c.c06c.e265  ARPA   Vlan1
    Internet  192.168.0.55            0   0026.1823.9a1b  ARPA   Vlan1
    Internet  192.168.2.192         170   7071.bc37.9d57  ARPA   Vlan1
    Internet  192.168.1.54            0   0026.1823.9cc9  ARPA   Vlan1
    Internet  192.168.1.70            0   7071.bc50.0d68  ARPA   Vlan1
    Internet  192.168.2.89            0   7071.bc51.c6e8  ARPA   Vlan1
    Internet  192.168.1.190           0   7071.bc51.c5c6  ARPA   Vlan1
    Internet  192.168.0.161           0   7071.bc50.0b53  ARPA   Vlan1
    Internet  192.168.0.113          26   0026.1823.9aa2  ARPA   Vlan1
    Internet  192.168.2.80          169   0023.54fa.52cf  ARPA   Vlan1
    Internet  192.168.2.174           0   0023.54fa.57f4  ARPA   Vlan1
    Internet  192.168.2.31            0   7071.bc37.9dff  ARPA   Vlan1
    Internet  192.168.3.18            0   0026.1823.9a97  ARPA   Vlan1
    Internet  192.168.0.136           0   0026.180a.73b1  ARPA   Vlan1
    Internet  192.168.1.53            0   001f.c6f7.decf  ARPA   Vlan1
    Internet  192.168.0.119           0   7071.bc07.3355  ARPA   Vlan1
    Internet  192.168.1.28            0   7071.bc50.0eb9  ARPA   Vlan1
    Internet  192.168.0.56            0   14da.e968.cbc7  ARPA   Vlan1
    Internet  192.168.0.35            0   14da.e968.cf91  ARPA   Vlan1
    Internet  192.168.0.219           0   7071.bc07.3363  ARPA   Vlan1
    Internet  192.168.1.81            0   0023.54fa.54ba  ARPA   Vlan1
    Internet  192.168.0.123           0   0026.1823.9a38  ARPA   Vlan1
    Internet  192.168.2.121         169   001c.c06c.d6cf  ARPA   Vlan1
    Internet  192.168.2.191           0   0023.54fa.59a6  ARPA   Vlan1
    Internet  192.168.2.122           0   c860.0088.6d9a  ARPA   Vlan1
    Internet  192.168.1.235           0   0026.1823.9a3d  ARPA   Vlan1
    Internet  192.168.1.242           0   0026.1823.9a31  ARPA   Vlan1
    Internet  192.168.2.202           0   0026.186e.ad4a  ARPA   Vlan1
    Internet  192.168.2.72            4   7071.bc50.0ccb  ARPA   Vlan1
    Internet  192.168.0.92            0   0026.188f.2584  ARPA   Vlan1
    Internet  192.168.0.3             0   0023.54fa.57ee  ARPA   Vlan1
    Internet  192.168.1.69            0   0026.1823.9a09  ARPA   Vlan1
    Internet  192.168.0.103           0   0026.180a.720f  ARPA   Vlan1
    Internet  192.168.0.183           0   7071.bc50.0cd8  ARPA   Vlan1
    Internet  192.168.2.7           170   14da.e968.c7a3  ARPA   Vlan1
    Internet  192.168.0.179           0   0023.54fa.516f  ARPA   Vlan1
    Internet  192.168.0.227           0   0026.185c.a625  ARPA   Vlan1
    Internet  192.168.1.7             0   7071.bc07.3300  ARPA   Vlan1
    Internet  192.168.0.221           0   0026.180a.72bd  ARPA   Vlan1
    Internet  192.168.0.222           0   0023.54fa.53b4  ARPA   Vlan1
    Internet  192.168.0.166           0   0026.1823.9adf  ARPA   Vlan1
    Internet  192.168.1.29            0   0023.54fa.53ef  ARPA   Vlan1
    Internet  192.168.1.252           0   001c.c06c.e017  ARPA   Vlan1
    Internet  192.168.0.4           170   14da.e968.d0b7  ARPA   Vlan1
    Internet  192.168.0.42            0   14da.e968.d032  ARPA   Vlan1
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  192.168.0.129           0   0026.180a.7301  ARPA   Vlan1
    Internet  192.168.0.11            0   0026.1823.9d94  ARPA   Vlan1
    Internet  192.168.0.177           0   7071.bc07.33dc  ARPA   Vlan1
    Internet  192.168.1.224           0   7071.bc07.3779  ARPA   Vlan1
    Internet  192.168.0.155           0   0023.54fa.57ef  ARPA   Vlan1
    Internet  192.168.1.232           0   7071.bc07.36ab  ARPA   Vlan1
    Internet  192.168.1.9            72   7071.bc07.32f7  ARPA   Vlan1
    Internet  192.168.1.165           0   7071.bc37.9e0b  ARPA   Vlan1
    Internet  192.168.0.128           0   0026.1823.9a1d  ARPA   Vlan1
    Internet  192.168.2.41          170   7071.bc51.c713  ARPA   Vlan1
    Internet  192.168.2.128           0   7071.bc50.0d76  ARPA   Vlan1
    Internet  192.168.1.4             0   7071.bc50.1471  ARPA   Vlan1
    Internet  192.168.1.37            0   14da.e968.cf8e  ARPA   Vlan1
    Internet  192.168.2.68            0   001c.c06c.d6c5  ARPA   Vlan1
    Internet  192.168.0.174           0   7071.bc07.3375  ARPA   Vlan1
    Internet  192.168.1.88           18   0026.180a.720a  ARPA   Vlan1
    Internet  192.168.0.236           0   0026.1823.9a08  ARPA   Vlan1
    Internet  192.168.0.142           0   0026.180a.7351  ARPA   Vlan1
    Internet  192.168.1.216           0   7071.bc07.32fb  ARPA   Vlan1
    Internet  192.168.2.203           0   0026.1823.9aaa  ARPA   Vlan1
    Internet  192.168.1.178         157   0026.188f.21ef  ARPA   Vlan1
    Internet  192.168.2.38            0   0026.1823.9a13  ARPA   Vlan1
    Internet  192.168.0.26            0   7071.bc06.2677  ARPA   Vlan1
    Internet  192.168.3.27            0   7071.bc50.0bdf  ARPA   Vlan1
    Internet  192.168.2.73            0   7071.bc50.0e79  ARPA   Vlan1
    Internet  192.168.1.231           0   7071.bc07.3304  ARPA   Vlan1
    Internet  192.168.2.28            0   7071.bc07.371e  ARPA   Vlan1
    Internet  192.168.0.78            0   7071.bc07.32f4  ARPA   Vlan1
    Internet  192.168.2.46            0   7071.bc50.0bf3  ARPA   Vlan1
    Internet  192.168.1.57            0   0026.188f.210b  ARPA   Vlan1
    Internet  192.168.2.88            0   7071.bc51.c81c  ARPA   Vlan1
    Internet  192.168.2.13            0   14da.e968.cb6a  ARPA   Vlan1
    Internet  192.168.1.221           0   7071.bc06.25e2  ARPA   Vlan1
    Internet  192.168.1.225           0   0023.54fa.59cc  ARPA   Vlan1
    Internet  192.168.0.25            0   14da.e968.c70e  ARPA   Vlan1
    Internet  192.168.2.201         170   7071.bc51.c89a  ARPA   Vlan1
    Internet  192.168.1.191           0   0019.d1e4.5634  ARPA   Vlan1
    Internet  192.168.1.60            0   7071.bc50.140c  ARPA   Vlan1
    Internet  192.168.3.159           0   7071.bc50.0e07  ARPA   Vlan1
    Internet  192.168.1.166           0   001c.c06c.d6ae  ARPA   Vlan1
    Internet  192.168.2.195           0   0023.54fa.51c6  ARPA   Vlan1
    Internet  192.168.1.159           0   0023.54fa.512a  ARPA   Vlan1
    Internet  192.168.2.106           0   0026.185c.a63c  ARPA   Vlan1
    Internet  192.168.3.16            0   7071.bc30.b182  ARPA   Vlan1
    Internet  192.168.2.105           0   0026.1831.c581  ARPA   Vlan1
    Internet  192.168.1.39            0   0023.54fa.59fc  ARPA   Vlan1
    Internet  192.168.0.160           0   7071.bc50.0e31  ARPA   Vlan1
    Internet  192.168.2.45            0   0021.7002.6a0e  ARPA   Vlan1
    Internet  192.168.0.21            0   7071.bc50.0dff  ARPA   Vlan1
    Internet  192.168.0.181           0   7071.bc50.0e8a  ARPA   Vlan1
    Internet  192.168.1.186           0   0023.54fa.5a1c  ARPA   Vlan1
    Internet  192.168.2.158           0   a41f.7253.85ce  ARPA   Vlan1
    Protocol  Address        &nb

  • MAC flap usually occurs when there is loop in layer two switches. Please check for your STP configuration or try shutting the redundant link down.

    Hope this helps!

  • Here's your problem.

    C 192.168.0.0/20 is directly connected, Vlan1

     

    Tied up now. Will try to come back later.

  • Hi,

    How many hosts approximatelly are connected to this single broadcast domain?

    Have you change any of the arp/mac timers on your switch?

  • If you are getting mac flaps it seems to me it would be a L2 problem possible bridge loop.  Maybe a dumb hub /switch in the network or LACP on one one side and misconfiged on the other?

    Can you trace down the source the the MAC being learned/relearned to a singe switch?  Try unplugging one of the ports that is getting the mac flap and see if that resolves the issue.  If not trace it back to the problem switch.  

    Example

    sw1 ------sw2----sw3----dumb hub (no stp)

    |                                   |

    |---------sw4----sw5------sw6

    In this case you would see mac learned/relearned (so many per second) on all the switches in this path.  On sw1 you will see mac flaps on the port facing sw2 and sw4.  On sw4 you will see it on the ports facing sw5 and sw1.  ETC.  

    On Sw1 when you remove the port facing Sw2 or Sw4 the problem should go away immediatly, but users on the dowstream would not be able to connect.  So if we remove Sw2 and the problem goes away plug it back in and remove the dowstream port to sw3.  Keep working your way back until you find the dumb hub and remove it.  

  • Hi,

    How many hosts approximatelly are connected to this single broadcast domain?

    Have you change any of the arp/mac timers on your switch?

     

    I was thinking along the same line. What I see here is a giant flat layer II network with 700+ hosts on the same vlan. Based on the information posted, it apears the arp aging timer is set to 180 mins instead of the 4 hours default value seen on Cisco switches. What this means is that all the intra-vlan communications -which could be a lot- is done through switching as opposed to routing. This inevitably will result in tons of arping. And the fast aging of arp entries just made things a lot worse. 

    Of course, the occasional MAC flapping will add CPU cycles too. But they won't show up on the "Arp Input" line though.

     

    Just my 2c

     

    +++ copied from original post

    CPU utilization for five seconds: 99%/12%; one minute: 99%; five minutes: 99%
     PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
      12   111438973    18587995       5995 44.47% 43.88% 43.96%   0 ARP Input

     

     

  • Hi,

     

    ITs not quite a flat later 2 network as host are on vlan 1 server, printers, phones, wifi and managment all have there own vlan. I have check the stp stuff and tthe correct device is root for the netowrk and the correct port are fowarding and blocking. I used "ip accounting mac-address" on interface vlan 1to see to top mac address that are talking and shut them down. This seems to have had some effect but was only tempory. I am sure I am getting alot of arp messages

    ARP statistics:
      Rcvd: 175893523 requests, 16526 replies, 1524 reverse, 0 other
      Sent: 703952 requests, 8392279 replies (42155 proxy), 0 reverse
      Drop due to input queue full: 4453836

    Vlan1
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                          IP   31557843 6047919076    3098013  240366653
                         ARP  175531088 10531865280    8307873  498472380
    Vlan10
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                          IP     135926   12108680     116272   10696446
                         ARP      15202     912124       4196     251760
    Vlan20
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                         ARP         51       3060          2        120
    Vlan30
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                       Other         78       4762          0          0
                          IP     299939   40698661     138852   16310438
               Spanning Tree          1         97          0          0
                         ARP     226641   13598460     129423    7765380
    Vlan40
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                          IP     305934   30987832     200110   21887686
                         ARP       2962     177720       2034     122040
    Vlan50
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                          IP     119934   17060183     116516   10715825
                         ARP      25196    1511760     550379   33022740
    Vlan60
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                       Other       1524      91440          0          0
                          IP     479094   71684264    1674193  205206465
                         ARP     217260   13035600      90270    5416200

     

    VLAN0001
      Spanning tree enabled protocol rstp
      Root ID    Priority    16385
                 Address     6c20.5660.a900
                 This bridge is the root
                 Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

    but am not sure why or where its coming from. There are  unmanaged switches in the network which were removed and this did not slove the issue.

  • also forgot to add that the arp age has not been changed and should be the default

  • peetypeety ✭✭✭

    Shrink your L2 domain.  /20 is very large.  Subdivide and conquer.

    Don't use VLAN1 for production traffic.  Pick any other VLAN, preferably several and split this up (as mentioned above).

    Is your STP topology reasonable and well-designed?  Is it even designed at all, or pure luck?  Is every switch running RSTP?

     

  • Hi,

     

    I have been recommending uses vlans per area so give subnets no bigger then a /24 for nearly  11 months. The stp topology is well-designed and I am sure this is not causing the issues. As a test all the redudency was taken off so there are no loops in the network and this issues still continued with no signs of improvement. The basic network has 2 core switches. The first switch( the one with the hi cpu) is set a root for all the vlan using a vlaue and not the spanning root command. Also all the interface have been check accross the network to see if everything is fowarding or blocking correctly and this has been the case. The network is running rapid spanning tree with 1 instance per vlan (rapid-pvst+). All trunk links run udld and all access ports are defined a  Edge P2p ports and have bpdu guard. Every switch had config redone and was tested and had been working fine for months.

     

    my personal view is this is a faulty device and/or a infected pc. I have made a plan to set up some vlans in the coming days and will see if this has an effect on the network.

  • peetypeety ✭✭✭

    The first switch( the one with the hi cpu) is set a root for all the vlan using a vlaue and not the spanning root command.

    FYI, the spanning root command just updates the config with "a value".

    What do you see with 'sh spann | i gy changes'?

    What STP features have you deployed to further stabilize the network?

    What debugs have you left running into your logging buffer in order to be certain it's not an STP problem?

  • As a test all the redudency was taken off so there are no loops in the network and this issues still continued with no signs of improvement

    Wait, so MAC flaps continued when you removed all redundancy?

  • I've seen MAC flaps with Q-in-Q.  In case you have that configured.

  • I saw the word "wifi" mentioned earlier. Mac flapping is pretty common on networks with wireless clients. It happens when mobile users are roaming from AP to AP. That in itself should not be a problem. 

  • Hi,

     

    Yes there are still mac flaps with no redundency at all and no unmanaged switch/hubs. All the mac flaps are on vlan 1. There is not q-in-q config at all. Wifi could have been a possible issue but I moved all wifi connection (controlers and AP's) to its own vlan and there are no mac flaps on that vlan. Going thought diagnostics I saw something that I could not explane. I saw a mac flap bewteen a port that was a trunk link and a port that was connected to a pc. Thinking this was strange I checked the port connected to the pc........



    Vlan    Mac Address       Type        Ports
    ----    -----------       --------    -----
       1    001c.c06c.d620    DYNAMIC     Gi1/1/3
       1    001c.c06c.d694    DYNAMIC     Gi1/1/3
       1    001c.c06c.d6ac    DYNAMIC     Gi1/1/3
       1    001c.c06c.d6e3    DYNAMIC     Gi1/1/3
       1    001c.c06c.d78c    DYNAMIC     Gi1/1/3
       1    001c.c06c.d7fc    DYNAMIC     Gi1/1/3
       1    001c.c06c.d879    DYNAMIC     Gi1/1/3
       1    0023.54fa.512a    DYNAMIC     Gi1/1/3
       1    0023.54fa.561b    DYNAMIC     Gi1/1/3
       1    0024.8c63.83b3    DYNAMIC     Gi1/1/3
       1    0024.8c8f.94a1    DYNAMIC     Gi1/1/3
       1    0026.180a.7025    DYNAMIC     Gi1/1/3
       1    0026.180a.7045    DYNAMIC     Gi1/1/3
       1    0026.180a.7205    DYNAMIC     Gi1/1/3
       1    0026.180a.7294    DYNAMIC     Gi1/1/3
       1    0026.180a.72ad    DYNAMIC     Gi1/1/3
       1    0026.180a.72bd    DYNAMIC     Gi1/1/3
       1    0026.180a.7328    DYNAMIC     Gi1/1/3
       1    0026.1823.99e4    DYNAMIC     Gi1/1/3
       1    0026.1823.99fa    DYNAMIC     Gi1/1/3
       1    0026.1823.9a09    DYNAMIC     Gi1/1/3
       1    0026.1823.9a8e    DYNAMIC     Gi1/1/3
       1    0026.1823.9a94    DYNAMIC     Gi1/1/3
       1    0026.1823.9f66    DYNAMIC     Gi1/1/3
       1    0026.1823.9f6e    DYNAMIC     Gi1/1/3
       1    0026.185c.a646    DYNAMIC     Gi1/1/3
       1    0026.185c.a9c9    DYNAMIC     Gi1/1/3
       1    0026.186e.ad4a    DYNAMIC     Gi1/1/3
       1    0026.188f.242a    DYNAMIC     Gi1/1/3
       1    14da.e968.c829    DYNAMIC     Gi1/1/3
       1    14da.e968.ca47    DYNAMIC     Gi1/1/3
       1    14da.e968.cfb4    DYNAMIC     Gi1/1/3
       1    60a4.4c64.8b1d    DYNAMIC     Gi1/1/3
       1    7071.bc06.24d1    DYNAMIC     Gi1/1/3
       1    7071.bc06.2525    DYNAMIC     Gi1/1/3
       1    7071.bc06.2641    DYNAMIC     Gi1/1/3
       1    7071.bc07.32fb    DYNAMIC     Gi1/1/3
       1    7071.bc07.3313    DYNAMIC     Gi1/1/3
       1    7071.bc07.36ad    DYNAMIC     Gi1/1/3
       1    7071.bc07.36f7    DYNAMIC     Gi1/1/3
       1    7071.bc50.0cde    DYNAMIC     Gi1/1/3
       1    7071.bc50.0f15    DYNAMIC     Gi1/1/3
       1    7071.bc51.c7a9    DYNAMIC     Gi1/1/3
       1    90e6.ba17.b2a0    DYNAMIC     Gi1/1/3
       1    90e6.ba17.b69b    DYNAMIC     Gi1/1/3
       1    a41f.7286.39e8    DYNAMIC     Gi1/1/3
       1    a41f.7286.3d19    DYNAMIC     Gi1/1/3
    Total Mac Addresses for this criterion: 47

     

    47 mac address from a PC? Virus checked and spyware checked all clean rebooted the pc and learnt 1 mac address. there are no virtual interface or and additional config on the pc. after around 10 minutes I start learning multiple mac address on that same port again.

  • Vlan1
                    Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                          IP   31557843 6047919076    3098013  240366653
                         ARP  175531088 10531865280    8307873  498472380

    Your arp traffic is off the chart. There might be something else going on here. I'd hook up a network sniffer and try to find out where all those arp requests are coming from. Perhaps some virus or malicious program? 

  • Do you have wireless clients on this VLAN? MAC flaps could be a consequence of your clients roaming between APs, so sometimes it is not a problem at all, just a sign that the clients are roaming through your facility, which is normal.

  • Hi,

     

    Yes there are still mac flaps with no redundency at all and no unmanaged switch/hubs. All the mac flaps are on vlan 1. There is not q-in-q config at all. Wifi could have been a possible issue but I moved all wifi connection (controlers and AP's) to its own vlan and there are no mac flaps on that vlan. Going thought diagnostics I saw something that I could not explane. I saw a mac flap bewteen a port that was a trunk link and a port that was connected to a pc. Thinking this was strange I checked the port connected to the pc........



    Vlan    Mac Address       Type        Ports
    ----    -----------       --------    -----
       1    001c.c06c.d620    DYNAMIC     Gi1/1/3
       1    001c.c06c.d694    DYNAMIC     Gi1/1/3
       1    001c.c06c.d6ac    DYNAMIC     Gi1/1/3
       1    001c.c06c.d6e3    DYNAMIC     Gi1/1/3
       1    001c.c06c.d78c    DYNAMIC     Gi1/1/3
       1    001c.c06c.d7fc    DYNAMIC     Gi1/1/3
       1    001c.c06c.d879    DYNAMIC     Gi1/1/3
       1    0023.54fa.512a    DYNAMIC     Gi1/1/3
       1    0023.54fa.561b    DYNAMIC     Gi1/1/3
       1    0024.8c63.83b3    DYNAMIC     Gi1/1/3
       1    0024.8c8f.94a1    DYNAMIC     Gi1/1/3
       1    0026.180a.7025    DYNAMIC     Gi1/1/3
       1    0026.180a.7045    DYNAMIC     Gi1/1/3
       1    0026.180a.7205    DYNAMIC     Gi1/1/3
       1    0026.180a.7294    DYNAMIC     Gi1/1/3
       1    0026.180a.72ad    DYNAMIC     Gi1/1/3
       1    0026.180a.72bd    DYNAMIC     Gi1/1/3
       1    0026.180a.7328    DYNAMIC     Gi1/1/3
       1    0026.1823.99e4    DYNAMIC     Gi1/1/3
       1    0026.1823.99fa    DYNAMIC     Gi1/1/3
       1    0026.1823.9a09    DYNAMIC     Gi1/1/3
       1    0026.1823.9a8e    DYNAMIC     Gi1/1/3
       1    0026.1823.9a94    DYNAMIC     Gi1/1/3
       1    0026.1823.9f66    DYNAMIC     Gi1/1/3
       1    0026.1823.9f6e    DYNAMIC     Gi1/1/3
       1    0026.185c.a646    DYNAMIC     Gi1/1/3
       1    0026.185c.a9c9    DYNAMIC     Gi1/1/3
       1    0026.186e.ad4a    DYNAMIC     Gi1/1/3
       1    0026.188f.242a    DYNAMIC     Gi1/1/3
       1    14da.e968.c829    DYNAMIC     Gi1/1/3
       1    14da.e968.ca47    DYNAMIC     Gi1/1/3
       1    14da.e968.cfb4    DYNAMIC     Gi1/1/3
       1    60a4.4c64.8b1d    DYNAMIC     Gi1/1/3
       1    7071.bc06.24d1    DYNAMIC     Gi1/1/3
       1    7071.bc06.2525    DYNAMIC     Gi1/1/3
       1    7071.bc06.2641    DYNAMIC     Gi1/1/3
       1    7071.bc07.32fb    DYNAMIC     Gi1/1/3
       1    7071.bc07.3313    DYNAMIC     Gi1/1/3
       1    7071.bc07.36ad    DYNAMIC     Gi1/1/3
       1    7071.bc07.36f7    DYNAMIC     Gi1/1/3
       1    7071.bc50.0cde    DYNAMIC     Gi1/1/3
       1    7071.bc50.0f15    DYNAMIC     Gi1/1/3
       1    7071.bc51.c7a9    DYNAMIC     Gi1/1/3
       1    90e6.ba17.b2a0    DYNAMIC     Gi1/1/3
       1    90e6.ba17.b69b    DYNAMIC     Gi1/1/3
       1    a41f.7286.39e8    DYNAMIC     Gi1/1/3
       1    a41f.7286.3d19    DYNAMIC     Gi1/1/3
    Total Mac Addresses for this criterion: 47

     

    47 mac address from a PC? Virus checked and spyware checked all clean rebooted the pc and learnt 1 mac address. there are no virtual interface or and additional config on the pc. after around 10 minutes I start learning multiple mac address on that same port again.

     

    Interesting. There definitely is more stuff going on here than meets the eye. Sounds like virus. But I would manually trace the cable on g1/1/3 to make sure it's not connected back to some other switch thereby forming a L2 loop. (Temporarily) kill the port may be a good idea too if that's feasible. 

  • The basic network has 2 core switches. The first switch( the one with the hi cpu) is set a root for all the vlan using a vlaue and not the spanning root command.

    So only one of the core switches is having high cpu? What do arp stats on the second core switch look like? Do they match that on the first switch? 

  • Interesting. There definitely is more stuff going on here than meets the eye. Sounds like virus. But I would manually trace the cable on g1/1/3 to make sure it's not connected back to some other switch thereby forming a L2 loop. (Temporarily) kill the port may be a good idea too if that's feasible. 

    Definitely worth double-checking the cabling.

    You could also use port-security on that port with a very limited number of MAC addresses.

  • Hi,

    I am sure there is something weard going on. I have done a wireshark capture and can see different computers on the network arping for eachother althought they should not be. I have been asured ther are no viruses or programs that caused caused this althought I am not so sure.

    The port 1/1/3 was set up by me so I can have contivity to the core switch as I shutdown the connections to all of ther ohter switches. It is 100% granted to be the pc with no devicedc in the middle. Multicast was enabled on the switch and when removed looked like it helped but then the rest of then network was back its the same stroy. I put a section of desktops in its own vlan and ( around 70 desktops) and it looks like it has not really helped. I am thinking to talk everything off vlan 1 and test

    I also noticed other ports  on access swithces that had larg numbers of mac address (100+) to ports that were connected to pc's. I will have to do alot more tshooting to find the cause.

  • The port 1/1/3 was set up by me so I can have contivity to the core switch as I shutdown the connections to all of ther ohter switches.

    If this is true, setup a port-security on that port to only allow your MAC Address and then set the violation mode to restrict. As soon as you do that, you will keep your connection up and running and then if all the other MACs will fail. A lot of things will start to have problems and then you will see what's going on.

  • 47 mac address from a PC?

    Do know the manufacturers of the PCs - you OUIs from -

    001cc0 - Intel

    002354 - ASUSTek COMPUTER INC

    00248c - As above

    002618 - As above

    14dae9 - As above

    60a44c - As above

    90e6ba - As above

    7071bc - PEGATRON CORPORATION

    a41f72 - DELL INC

    Have you got a hub or switch on the end of this port? If not can you share you Wireshark traces?  Have you captured other data other than ARP?

    Do you have PCs from these manufacturers?

     

     

     

     

  • Could also be a misconfigured port-channel.  You will definitely see mac-flaps if you have PC configured at one end, but not at other - this will not show up in a valid STP topology, as the topology is still valid (root bridges, stp priorities etc), but you'll still get a loop.

  • I had a similar problem on my network few years back. Do you have any GLBP/HSRP running? If so make sure the active forwarding device is the spanning tree root. You might still have looping traffic through unexpected paths(.e.g. at access layer). You need smaller subnets e.g. /24 and move everything off vlan 1 and prune vlan one off trunk links. Vlan 1 is carries control traffic and when you have spanning-tree noise or other protocol noise it just makes things worse with everything else right there. 

    Make sure proxy arp is disabled on router interfaces as hosts with misconfigured IP/Subnet masks can trigger extra arp noise with the router responding on behalf of the hosts "gateway". Also ensure there is no other device connected that has proxy-arp enabled-in show ip arp output look out for many MAC addresses resolving to the same IP Address.

     

  • Hi,

    We are aware of the mac address and know some of the pc's its coming from.
    Port security is not an option at the moment as pc get moved around and we can
    have them shutdown. Also there are around 20-30 access switches so it would be
    a time consuming job. This can be scheduled in but won’t be for a while.

    Most of our PC are RM and they have some components from these manufactures.
    We did look up the mac first thing when we saw the issue.

    Miss configured port channels do cause mac flaps but in this case I check
    this first and they are all fine (switch and server side). I believe the mac
    flaps are the cause of desktops advertising multiple mac address to the
    switches. I noticed at least 4 ports on different switches that are doing the
    same thing and learn the same mac address on.

    We are running HSRP and it is the root bridge that is master for HSRP. I
    disabled proxy arp when I got this issue as a first step. I have managed to
    reduce the overall cpu to 70-60% and arp input to 20-25% after powering down
    all computers and then reloading all switches. The source may have been a
    server which we are investigating. The arp is still very high for my liking and
    we are continuing to see what we can do. Cisco tac have been involved and they
    have not had any luck working out what is going on.

    Dynamic arp was an idea but can’t do it as some of the edge switches are
    2950 and its not supported. A big thanks to everyone for your grate ideas still
    open to more.

Sign In or Register to comment.