Help with ARP issue
Hi,
I have a 3750x switch that have very high CPU. The cause of this seems to be the ARP Input with is running at around 40%-60% and the cpu is running at around 99%. What can I do to find out that is causing the ARP Input to go so high?
CPU utilization for five seconds: 99%/12%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 111438973 18587995 5995 44.47% 43.88% 43.96% 0 ARP Input
174 59541847 5198737 11453 22.39% 23.47% 23.62% 0 Hulc LED Process
221 7253246 6147816 1179 4.95% 4.25% 4.10% 0 IP Input
86 5459437 1100349 4961 1.59% 1.47% 1.54% 0 RedEarth Tx Mana
85 3448684 1453278 2373 1.27% 1.04% 1.07% 0 RedEarth I2C dri
any ideas would be much appreciated
Comments
You might have something like the following in your configuration :
Ip route 0.0.0.0 0.0.0.0 f1/0
In this case, the router will arp for every destination of which the next hop is unknown.
Thank you for you reply, There are no configured static routes as it is using OSPF for routeing.
Firstly, check if you have loop scenario.
Is it operating in layer 2 or layer 3 mode?
Interesting. Can you paste "sh arp" and "sh ip route"?
The switch is oprating at both layer 2 and layer 3. There are around 20 switches connected back to this 3750x stack. I have noticed a number of mac flaps as well.
I have included arp for vlan 1 as this is the problem vlan.
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.15.224 - 6c20.5660.a940 ARPA Vlan1
Internet 192.168.0.141 0 0023.54fa.598d ARPA Vlan1
Internet 192.168.0.208 0 0026.180a.733f ARPA Vlan1
Internet 192.168.1.189 0 0026.185c.a63a ARPA Vlan1
Internet 192.168.1.138 0 7071.bc06.2682 ARPA Vlan1
Internet 192.168.0.143 0 0023.54fa.5997 ARPA Vlan1
Internet 192.168.0.151 0 7071.bc06.2630 ARPA Vlan1
Internet 192.168.1.34 0 90e6.ba17.b2a0 ARPA Vlan1
Internet 192.168.1.177 0 00e0.4c68.0050 ARPA Vlan1
Internet 192.168.15.26 0 f8d1.1100.133f ARPA Vlan1
Internet 192.168.2.130 0 001c.c06c.d587 ARPA Vlan1
Internet 192.168.1.204 0 001c.c06c.d8b6 ARPA Vlan1
Internet 192.168.0.83 0 14da.e968.d143 ARPA Vlan1
Internet 192.168.15.10 0 60a4.4c64.8b1d ARPA Vlan1
Internet 192.168.14.152 0 0014.fd11.843e ARPA Vlan1
Internet 192.168.15.11 0 60a4.4c64.8c21 ARPA Vlan1
Internet 192.168.0.9 0 902b.3440.5ad5 ARPA Vlan1
Internet 192.168.0.77 0 0026.180a.7039 ARPA Vlan1
Internet 192.168.1.41 0 7071.bc07.337d ARPA Vlan1
Internet 192.168.3.31 2 6805.ca05.7418 ARPA Vlan1
Internet 192.168.14.31 1 0011.4335.39c4 ARPA Vlan1
Internet 192.168.0.200 0 0019.d1e4.5717 ARPA Vlan1
Internet 192.168.15.25 0 3cd9.2b5d.2cc3 ARPA Vlan1
Internet 192.168.15.30 0 f8d1.1102.ef7c ARPA Vlan1
Internet 192.168.15.101 6 0090.0b18.f872 ARPA Vlan1
Internet 192.168.15.100 169 0090.0b18.f870 ARPA Vlan1
Internet 192.168.14.3 0 0001.8021.1cba ARPA Vlan1
Internet 192.168.15.225 1 6c20.566f.bcc0 ARPA Vlan1
Internet 192.168.14.26 1 0003.4711.5043 ARPA Vlan1
Internet 192.168.1.5 0 90e6.ba17.b8f8 ARPA Vlan1
Internet 192.168.14.30 0 0024.21a8.0960 ARPA Vlan1
Internet 192.168.14.151 0 902b.3440.0a2d ARPA Vlan1
Internet 192.168.1.205 0 001c.c06c.d576 ARPA Vlan1
Internet 192.168.1.75 0 001c.c06c.d858 ARPA Vlan1
Internet 192.168.3.17 0 a41f.7286.4779 ARPA Vlan1
Internet 192.168.0.7 0 f46d.0494.5349 ARPA Vlan1
Internet 192.168.1.223 0 7071.bc07.335c ARPA Vlan1
Internet 192.168.0.173 0 7071.bc50.0b4d ARPA Vlan1
Internet 192.168.0.48 0 0026.1823.9e4d ARPA Vlan1
Internet 192.168.0.97 0 0026.185c.a8ea ARPA Vlan1
Internet 192.168.2.99 0 001c.c06c.d5f2 ARPA Vlan1
Internet 192.168.2.95 0 001c.c06c.d878 ARPA Vlan1
Internet 192.168.2.97 4 001c.c06c.d782 ARPA Vlan1
Internet 192.168.1.116 0 7071.bc06.24b9 ARPA Vlan1
Internet 192.168.0.233 0 0026.180a.72c1 ARPA Vlan1
Internet 192.168.2.58 0 001c.c06c.d802 ARPA Vlan1
Internet 192.168.0.220 0 0026.180a.720c ARPA Vlan1
Internet 192.168.2.87 0 001c.c06c.d6f5 ARPA Vlan1
Internet 192.168.1.187 0 7071.bc50.0e45 ARPA Vlan1
Internet 192.168.0.106 0 0026.185c.a60f ARPA Vlan1
Internet 192.168.1.73 0 0026.185c.a8ce ARPA Vlan1
Internet 192.168.1.79 0 0026.180a.7069 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.0.98 0 7071.bc07.36a7 ARPA Vlan1
Internet 192.168.1.11 0 001c.c06c.e296 ARPA Vlan1
Internet 192.168.0.81 0 0026.1823.9b3a ARPA Vlan1
Internet 192.168.2.66 0 001c.c06c.da67 ARPA Vlan1
Internet 192.168.0.203 0 001c.c06c.d8fb ARPA Vlan1
Internet 192.168.0.210 0 0026.180a.734d ARPA Vlan1
Internet 192.168.1.181 0 001c.c06c.dc1b ARPA Vlan1
Internet 192.168.1.45 0 001c.c06c.d808 ARPA Vlan1
Internet 192.168.0.157 0 0024.8c63.85ba ARPA Vlan1
Internet 192.168.1.167 0 001c.c06c.d6fd ARPA Vlan1
Internet 192.168.1.126 0 001c.c06c.e284 ARPA Vlan1
Internet 192.168.0.252 0 001c.c06c.d8d5 ARPA Vlan1
Internet 192.168.1.194 0 7071.bc51.c7a9 ARPA Vlan1
Internet 192.168.0.105 0 0026.180a.7359 ARPA Vlan1
Internet 192.168.0.218 0 0026.180a.7205 ARPA Vlan1
Internet 192.168.1.3 0 001f.c6f7.dfa5 ARPA Vlan1
Internet 192.168.1.63 0 001c.c06c.d78c ARPA Vlan1
Internet 192.168.1.8 0 001f.c6f7.dec2 ARPA Vlan1
Internet 192.168.0.19 0 001c.c06c.d5d5 ARPA Vlan1
Internet 192.168.1.243 0 7071.bc07.36a9 ARPA Vlan1
Internet 192.168.0.225 0 001c.c06c.d90f ARPA Vlan1
Internet 192.168.0.180 0 001c.c06c.d831 ARPA Vlan1
Internet 192.168.1.219 0 7071.bc07.3324 ARPA Vlan1
Internet 192.168.3.4 169 0026.180a.7188 ARPA Vlan1
Internet 192.168.0.206 169 7071.bc51.c89c ARPA Vlan1
Internet 192.168.2.109 0 001c.c06c.d68b ARPA Vlan1
Internet 192.168.0.8 0 001c.c06c.d7b0 ARPA Vlan1
Internet 192.168.1.152 169 001c.c06c.d7fc ARPA Vlan1
Internet 192.168.2.71 0 7071.bc50.0c3d ARPA Vlan1
Internet 192.168.1.13 0 0026.1823.99e5 ARPA Vlan1
Internet 192.168.1.144 0 001c.c06c.d5d9 ARPA Vlan1
Internet 192.168.1.250 0 7071.bc06.2529 ARPA Vlan1
Internet 192.168.1.59 0 0019.d1e4.569c ARPA Vlan1
Internet 192.168.2.117 86 60a4.4c64.1154 ARPA Vlan1
Internet 192.168.0.246 0 7071.bc37.9c42 ARPA Vlan1
Internet 192.168.0.109 0 0026.1823.9a1c ARPA Vlan1
Internet 192.168.1.23 0 0026.1823.99fd ARPA Vlan1
Internet 192.168.0.154 0 7071.bc50.0e4c ARPA Vlan1
Internet 192.168.0.175 0 7071.bc37.9d97 ARPA Vlan1
Internet 192.168.1.67 0 7071.bc37.9da5 ARPA Vlan1
Internet 192.168.0.49 0 0026.180a.7036 ARPA Vlan1
Internet 192.168.0.130 0 7071.bc50.0a89 ARPA Vlan1
Internet 192.168.1.52 0 90e2.ba40.1761 ARPA Vlan1
Internet 192.168.0.229 0 0023.54fa.59c5 ARPA Vlan1
Internet 192.168.1.36 0 0023.54fa.59cf ARPA Vlan1
Internet 192.168.0.211 0 0026.1823.9a1f ARPA Vlan1
Internet 192.168.2.0 0 001c.c065.dd9d ARPA Vlan1
Internet 192.168.3.5 0 0026.1823.9a36 ARPA Vlan1
Internet 192.168.0.39 0 0026.180a.7373 ARPA Vlan1
Internet 192.168.0.93 0 0026.185c.a8a1 ARPA Vlan1
Internet 192.168.0.144 0 7071.bc06.24a5 ARPA Vlan1
Internet 192.168.3.10 0 0026.1823.9cdc ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.0.31 0 14da.e968.d02a ARPA Vlan1
Internet 192.168.2.140 0 0023.54fa.59a7 ARPA Vlan1
Internet 192.168.1.62 0 001c.c06c.dfa1 ARPA Vlan1
Internet 192.168.0.114 0 0026.180a.7328 ARPA Vlan1
Internet 192.168.0.30 0 7071.bc06.2631 ARPA Vlan1
Internet 192.168.2.84 0 7071.bc07.36ad ARPA Vlan1
Internet 192.168.0.248 0 0026.1823.9b5d ARPA Vlan1
Internet 192.168.0.107 0 0026.1823.9ce2 ARPA Vlan1
Internet 192.168.0.249 0 0026.185c.a888 ARPA Vlan1
Internet 192.168.2.155 0 7071.bc30.b168 ARPA Vlan1
Internet 192.168.0.67 0 14da.e968.cb95 ARPA Vlan1
Internet 192.168.0.13 0 0026.180a.72fd ARPA Vlan1
Internet 192.168.1.111 0 001c.c06c.d7c6 ARPA Vlan1
Internet 192.168.2.138 0 0026.1823.99e4 ARPA Vlan1
Internet 192.168.0.240 0 0026.1823.9aad ARPA Vlan1
Internet 192.168.0.124 0 0026.1823.9a0e ARPA Vlan1
Internet 192.168.1.22 0 0026.184e.137f ARPA Vlan1
Internet 192.168.0.75 0 7071.bc37.9ded ARPA Vlan1
Internet 192.168.0.62 0 0026.180a.7215 ARPA Vlan1
Internet 192.168.2.96 0 001c.c06c.d6d5 ARPA Vlan1
Internet 192.168.0.22 0 0026.180a.7210 ARPA Vlan1
Internet 192.168.2.175 0 7071.bc50.1465 ARPA Vlan1
Internet 192.168.0.137 0 0026.180a.73bb ARPA Vlan1
Internet 192.168.0.145 0 7071.bc30.afe7 ARPA Vlan1
Internet 192.168.2.47 0 7071.bc50.0b73 ARPA Vlan1
Internet 192.168.1.218 0 7071.bc07.32f9 ARPA Vlan1
Internet 192.168.1.77 169 7071.bc07.36f5 ARPA Vlan1
Internet 192.168.0.239 0 0026.180a.7326 ARPA Vlan1
Internet 192.168.1.40 0 0026.185c.aaf8 ARPA Vlan1
Internet 192.168.0.149 0 7071.bc06.24a7 ARPA Vlan1
Internet 192.168.0.111 0 0023.54fa.5a3a ARPA Vlan1
Internet 192.168.0.96 0 001f.c6f7.ded1 ARPA Vlan1
Internet 192.168.0.243 0 0026.180a.7357 ARPA Vlan1
Internet 192.168.1.82 0 7071.bc50.0ee6 ARPA Vlan1
Internet 192.168.0.244 142 7071.bc37.9cc1 ARPA Vlan1
Internet 192.168.0.190 0 001c.c06c.d81e ARPA Vlan1
Internet 192.168.2.238 0 14da.e968.c925 ARPA Vlan1
Internet 192.168.0.82 0 14da.e968.c8cf ARPA Vlan1
Internet 192.168.1.246 0 0026.1823.99ec ARPA Vlan1
Internet 192.168.1.161 0 7071.bc06.24f0 ARPA Vlan1
Internet 192.168.0.214 0 0026.1823.9a46 ARPA Vlan1
Internet 192.168.0.116 0 0024.8c8f.94a1 ARPA Vlan1
Internet 192.168.0.162 0 7071.bc07.32ff ARPA Vlan1
Internet 192.168.0.202 0 7071.bc07.3317 ARPA Vlan1
Internet 192.168.3.9 0 0026.180a.7045 ARPA Vlan1
Internet 192.168.0.122 0 0026.180a.7074 ARPA Vlan1
Internet 192.168.0.131 0 7071.bc07.3303 ARPA Vlan1
Internet 192.168.0.132 0 0026.1823.9c86 ARPA Vlan1
Internet 192.168.2.42 0 0026.1823.9a03 ARPA Vlan1
Internet 192.168.2.193 0 0023.54fa.59fe ARPA Vlan1
Internet 192.168.0.99 0 0026.1823.9c8c ARPA Vlan1
Internet 192.168.1.38 0 7071.bc50.0d74 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.91 169 0023.54fa.59a0 ARPA Vlan1
Internet 192.168.1.227 0 7071.bc06.24d5 ARPA Vlan1
Internet 192.168.0.85 0 14da.e968.d025 ARPA Vlan1
Internet 192.168.0.68 0 7071.bc06.2543 ARPA Vlan1
Internet 192.168.1.64 0 0026.1823.9ac8 ARPA Vlan1
Internet 192.168.0.170 0 7071.bc07.32f0 ARPA Vlan1
Internet 192.168.1.83 0 14da.e968.c829 ARPA Vlan1
Internet 192.168.1.222 0 0023.54fa.5a06 ARPA Vlan1
Internet 192.168.2.104 0 001c.c06c.d766 ARPA Vlan1
Internet 192.168.2.1 0 001c.c06c.d723 ARPA Vlan1
Internet 192.168.0.73 0 14da.e968.d026 ARPA Vlan1
Internet 192.168.0.10 0 14da.e968.c7f8 ARPA Vlan1
Internet 192.168.1.15 0 0026.180a.7294 ARPA Vlan1
Internet 192.168.1.226 0 7071.bc07.331b ARPA Vlan1
Internet 192.168.0.241 0 0026.1823.9ce5 ARPA Vlan1
Internet 192.168.0.167 0 1c6f.658f.470d ARPA Vlan1
Internet 192.168.1.72 0 7071.bc50.1449 ARPA Vlan1
Internet 192.168.0.153 0 7071.bc07.3353 ARPA Vlan1
Internet 192.168.0.43 0 0023.54fa.59e2 ARPA Vlan1
Internet 192.168.0.61 0 14da.e968.cc1a ARPA Vlan1
Internet 192.168.2.6 169 001c.c06c.d6df ARPA Vlan1
Internet 192.168.1.233 0 7071.bc06.2528 ARPA Vlan1
Internet 192.168.0.15 0 14da.e968.cb62 ARPA Vlan1
Internet 192.168.2.85 0 0023.54fa.52ef ARPA Vlan1
Internet 192.168.0.69 0 0019.d189.315a ARPA Vlan1
Internet 192.168.0.205 0 001c.c06c.d620 ARPA Vlan1
Internet 192.168.2.53 0 001c.c06c.d695 ARPA Vlan1
Internet 192.168.2.8 0 14da.e968.d178 ARPA Vlan1
Internet 192.168.2.173 0 0026.1823.9a93 ARPA Vlan1
Internet 192.168.0.52 0 14da.e968.cfa4 ARPA Vlan1
Internet 192.168.2.14 169 14da.e968.d029 ARPA Vlan1
Internet 192.168.3.2 0 0026.1823.99e9 ARPA Vlan1
Internet 192.168.1.119 0 14da.e968.c8dc ARPA Vlan1
Internet 192.168.2.240 0 0026.180a.7065 ARPA Vlan1
Internet 192.168.0.238 0 7071.bc51.c766 ARPA Vlan1
Internet 192.168.2.18 169 14da.e968.c826 ARPA Vlan1
Internet 192.168.0.53 0 0026.180a.72fb ARPA Vlan1
Internet 192.168.1.117 0 14da.e968.c82e ARPA Vlan1
Internet 192.168.1.105 0 001c.c06c.d8a6 ARPA Vlan1
Internet 192.168.2.24 0 001c.c06c.d794 ARPA Vlan1
Internet 192.168.2.48 0 0026.180a.7063 ARPA Vlan1
Internet 192.168.0.34 0 0026.180a.7355 ARPA Vlan1
Internet 192.168.0.138 0 0024.8c58.9abc ARPA Vlan1
Internet 192.168.2.60 0 0026.1823.9ab1 ARPA Vlan1
Internet 192.168.2.172 0 0026.1823.9a98 ARPA Vlan1
Internet 192.168.1.113 0 001c.c06c.d97b ARPA Vlan1
Internet 192.168.1.228 169 0026.1823.9ad5 ARPA Vlan1
Internet 192.168.0.91 0 0026.1823.9ab0 ARPA Vlan1
Internet 192.168.2.123 0 7071.bc06.2641 ARPA Vlan1
Internet 192.168.2.197 0 001c.c06c.d8f3 ARPA Vlan1
Internet 192.168.0.212 0 0026.1823.9acf ARPA Vlan1
Internet 192.168.0.102 0 0026.185c.aaf0 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.2.242 0 14da.e968.cc5b ARPA Vlan1
Internet 192.168.2.112 0 001c.c06c.d6e3 ARPA Vlan1
Internet 192.168.1.212 169 0024.8c63.83b3 ARPA Vlan1
Internet 192.168.0.232 0 001c.c06c.e223 ARPA Vlan1
Internet 192.168.0.58 0 14da.e968.cb8f ARPA Vlan1
Internet 192.168.2.93 0 001c.c06c.dc23 ARPA Vlan1
Internet 192.168.1.90 0 7071.bc50.12f4 ARPA Vlan1
Internet 192.168.0.193 0 14da.e968.ca22 ARPA Vlan1
Internet 192.168.1.31 0 7071.bc07.349f ARPA Vlan1
Internet 192.168.0.46 0 0026.180a.717a ARPA Vlan1
Internet 192.168.0.195 31 0026.180a.7293 ARPA Vlan1
Internet 192.168.2.102 0 001c.c06c.d67e ARPA Vlan1
Internet 192.168.0.24 0 001c.c06c.d737 ARPA Vlan1
Internet 192.168.2.5 169 14da.e968.d02b ARPA Vlan1
Internet 192.168.2.125 0 7071.bc51.c8b2 ARPA Vlan1
Internet 192.168.0.27 0 14da.e968.d185 ARPA Vlan1
Internet 192.168.0.72 0 14da.e968.c834 ARPA Vlan1
Internet 192.168.0.188 0 001c.c06c.d47c ARPA Vlan1
Internet 192.168.2.3 0 14da.e968.c7c9 ARPA Vlan1
Internet 192.168.0.254 42 0023.54fa.57f0 ARPA Vlan1
Internet 192.168.0.51 169 0026.1823.9cd4 ARPA Vlan1
Internet 192.168.2.29 0 001f.c6f7.decb ARPA Vlan1
Internet 192.168.0.32 0 14da.e968.cdac ARPA Vlan1
Internet 192.168.1.18 0 001c.c06c.d56d ARPA Vlan1
Internet 192.168.1.21 0 001c.c06c.e225 ARPA Vlan1
Internet 192.168.1.74 0 0026.180a.7059 ARPA Vlan1
Internet 192.168.1.98 0 7071.bc06.266b ARPA Vlan1
Internet 192.168.2.171 0 0026.1823.9a2f ARPA Vlan1
Internet 192.168.0.64 0 0026.1823.9a0d ARPA Vlan1
Internet 192.168.0.36 0 001c.c06c.d80a ARPA Vlan1
Internet 192.168.0.215 0 0026.180a.72a1 ARPA Vlan1
Internet 192.168.0.217 0 0026.185c.a5ef ARPA Vlan1
Internet 192.168.3.6 0 0026.1823.9b31 ARPA Vlan1
Internet 192.168.2.243 169 0026.1823.9cd3 ARPA Vlan1
Internet 192.168.0.95 0 0023.54fa.5979 ARPA Vlan1
Internet 192.168.0.139 0 14da.e968.d023 ARPA Vlan1
Internet 192.168.0.134 0 0026.1823.9a8e ARPA Vlan1
Internet 192.168.1.25 0 0026.1823.9acc ARPA Vlan1
Internet 192.168.0.223 0 0026.185c.a620 ARPA Vlan1
Internet 192.168.2.52 0 0026.1823.9a3e ARPA Vlan1
Internet 192.168.0.209 0 0026.1823.9a8a ARPA Vlan1
Internet 192.168.2.91 0 001c.c06c.dc33 ARPA Vlan1
Internet 192.168.0.89 0 90e6.ba17.bb11 ARPA Vlan1
Internet 192.168.0.2 0 001c.c06c.e04a ARPA Vlan1
Internet 192.168.0.172 0 001c.c06c.d79a ARPA Vlan1
Internet 192.168.2.126 0 7071.bc37.a620 ARPA Vlan1
Internet 192.168.0.71 0 0026.1823.9a9c ARPA Vlan1
Internet 192.168.2.170 0 0026.185c.a773 ARPA Vlan1
Internet 192.168.1.207 0 001c.c06c.d879 ARPA Vlan1
Internet 192.168.0.169 0 14da.e968.c8e1 ARPA Vlan1
Internet 192.168.2.116 169 001c.c06c.d64d ARPA Vlan1
Internet 192.168.1.249 0 7071.bc07.3364 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.2.244 0 0026.1823.9cc1 ARPA Vlan1
Internet 192.168.0.186 0 0026.180a.7307 ARPA Vlan1
Internet 192.168.0.135 0 0026.180a.7217 ARPA Vlan1
Internet 192.168.1.170 0 7071.bc07.32fe ARPA Vlan1
Internet 192.168.2.98 0 001c.c06c.d5e7 ARPA Vlan1
Internet 192.168.0.140 0 0026.180a.720b ARPA Vlan1
Internet 192.168.0.216 0 0026.1823.9bff ARPA Vlan1
Internet 192.168.1.248 0 0026.1823.99e8 ARPA Vlan1
Internet 192.168.2.27 0 90e6.ba17.b66c ARPA Vlan1
Internet 192.168.2.10 170 14da.e968.c7e1 ARPA Vlan1
Internet 192.168.0.191 0 0026.180a.72ad ARPA Vlan1
Internet 192.168.1.171 0 7071.bc50.0c98 ARPA Vlan1
Internet 192.168.1.237 0 7071.bc07.3347 ARPA Vlan1
Internet 192.168.2.124 0 7071.bc50.0f15 ARPA Vlan1
Internet 192.168.1.143 0 7071.bc50.1364 ARPA Vlan1
Internet 192.168.0.28 0 14da.e968.ca47 ARPA Vlan1
Internet 192.168.1.19 0 0026.1823.9c04 ARPA Vlan1
Internet 192.168.2.55 0 001c.c06c.dc04 ARPA Vlan1
Internet 192.168.1.236 0 0026.1823.9a22 ARPA Vlan1
Internet 192.168.0.110 0 0026.1823.9aac ARPA Vlan1
Internet 192.168.0.65 0 0023.54fa.59ec ARPA Vlan1
Internet 192.168.1.27 0 0023.54fa.57cf ARPA Vlan1
Internet 192.168.2.198 0 001c.c06c.d678 ARPA Vlan1
Internet 192.168.1.12 0 0023.54fa.5441 ARPA Vlan1
Internet 192.168.2.15 0 14da.e968.cb5a ARPA Vlan1
Internet 192.168.1.118 0 14da.e968.cbcd ARPA Vlan1
Internet 192.168.1.255 0 001c.c06c.d6d9 ARPA Vlan1
Internet 192.168.0.86 0 7071.bc07.3318 ARPA Vlan1
Internet 192.168.0.70 0 7071.bc07.3358 ARPA Vlan1
Internet 192.168.1.220 0 7071.bc06.267f ARPA Vlan1
Internet 192.168.3.3 0 0026.1823.9bca ARPA Vlan1
Internet 192.168.0.127 0 0026.1823.9a96 ARPA Vlan1
Internet 192.168.2.44 0 0026.180a.7020 ARPA Vlan1
Internet 192.168.0.194 0 14da.e968.d156 ARPA Vlan1
Internet 192.168.1.230 0 7071.bc06.2688 ARPA Vlan1
Internet 192.168.1.157 0 0026.1823.9a10 ARPA Vlan1
Internet 192.168.0.74 169 001c.c06c.dffa ARPA Vlan1
Internet 192.168.1.58 0 0019.d1e4.5691 ARPA Vlan1
Internet 192.168.1.20 0 0026.184e.137b ARPA Vlan1
Internet 192.168.0.245 0 7071.bc07.3357 ARPA Vlan1
Internet 192.168.1.229 0 7071.bc07.36f1 ARPA Vlan1
Internet 192.168.0.63 0 0026.180a.7295 ARPA Vlan1
Internet 192.168.1.43 0 7071.bc06.2501 ARPA Vlan1
Internet 192.168.0.84 0 14da.e968.c801 ARPA Vlan1
Internet 192.168.0.199 0 0026.1823.9c05 ARPA Vlan1
Internet 192.168.0.101 0 14da.e968.cc72 ARPA Vlan1
Internet 192.168.2.54 0 001c.c06c.d870 ARPA Vlan1
Internet 192.168.2.94 0 001c.c06c.d814 ARPA Vlan1
Internet 192.168.0.88 0 7071.bc50.1478 ARPA Vlan1
Internet 192.168.0.204 0 14da.e968.cb61 ARPA Vlan1
Internet 192.168.2.4 0 14da.e968.c755 ARPA Vlan1
Internet 192.168.1.26 169 0023.54fa.5522 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.192 0 001c.c06c.d694 ARPA Vlan1
Internet 192.168.1.32 0 7071.bc50.0e60 ARPA Vlan1
Internet 192.168.1.2 0 0023.54fa.5711 ARPA Vlan1
Internet 192.168.0.242 0 0026.180a.7146 ARPA Vlan1
Internet 192.168.2.127 0 7071.bc50.0eb5 ARPA Vlan1
Internet 192.168.0.165 0 0026.180a.7047 ARPA Vlan1
Internet 192.168.2.129 0 0026.1823.9a94 ARPA Vlan1
Internet 192.168.2.90 0 001c.c06c.e265 ARPA Vlan1
Internet 192.168.0.55 0 0026.1823.9a1b ARPA Vlan1
Internet 192.168.2.192 170 7071.bc37.9d57 ARPA Vlan1
Internet 192.168.1.54 0 0026.1823.9cc9 ARPA Vlan1
Internet 192.168.1.70 0 7071.bc50.0d68 ARPA Vlan1
Internet 192.168.2.89 0 7071.bc51.c6e8 ARPA Vlan1
Internet 192.168.1.190 0 7071.bc51.c5c6 ARPA Vlan1
Internet 192.168.0.161 0 7071.bc50.0b53 ARPA Vlan1
Internet 192.168.0.113 26 0026.1823.9aa2 ARPA Vlan1
Internet 192.168.2.80 169 0023.54fa.52cf ARPA Vlan1
Internet 192.168.2.174 0 0023.54fa.57f4 ARPA Vlan1
Internet 192.168.2.31 0 7071.bc37.9dff ARPA Vlan1
Internet 192.168.3.18 0 0026.1823.9a97 ARPA Vlan1
Internet 192.168.0.136 0 0026.180a.73b1 ARPA Vlan1
Internet 192.168.1.53 0 001f.c6f7.decf ARPA Vlan1
Internet 192.168.0.119 0 7071.bc07.3355 ARPA Vlan1
Internet 192.168.1.28 0 7071.bc50.0eb9 ARPA Vlan1
Internet 192.168.0.56 0 14da.e968.cbc7 ARPA Vlan1
Internet 192.168.0.35 0 14da.e968.cf91 ARPA Vlan1
Internet 192.168.0.219 0 7071.bc07.3363 ARPA Vlan1
Internet 192.168.1.81 0 0023.54fa.54ba ARPA Vlan1
Internet 192.168.0.123 0 0026.1823.9a38 ARPA Vlan1
Internet 192.168.2.121 169 001c.c06c.d6cf ARPA Vlan1
Internet 192.168.2.191 0 0023.54fa.59a6 ARPA Vlan1
Internet 192.168.2.122 0 c860.0088.6d9a ARPA Vlan1
Internet 192.168.1.235 0 0026.1823.9a3d ARPA Vlan1
Internet 192.168.1.242 0 0026.1823.9a31 ARPA Vlan1
Internet 192.168.2.202 0 0026.186e.ad4a ARPA Vlan1
Internet 192.168.2.72 4 7071.bc50.0ccb ARPA Vlan1
Internet 192.168.0.92 0 0026.188f.2584 ARPA Vlan1
Internet 192.168.0.3 0 0023.54fa.57ee ARPA Vlan1
Internet 192.168.1.69 0 0026.1823.9a09 ARPA Vlan1
Internet 192.168.0.103 0 0026.180a.720f ARPA Vlan1
Internet 192.168.0.183 0 7071.bc50.0cd8 ARPA Vlan1
Internet 192.168.2.7 170 14da.e968.c7a3 ARPA Vlan1
Internet 192.168.0.179 0 0023.54fa.516f ARPA Vlan1
Internet 192.168.0.227 0 0026.185c.a625 ARPA Vlan1
Internet 192.168.1.7 0 7071.bc07.3300 ARPA Vlan1
Internet 192.168.0.221 0 0026.180a.72bd ARPA Vlan1
Internet 192.168.0.222 0 0023.54fa.53b4 ARPA Vlan1
Internet 192.168.0.166 0 0026.1823.9adf ARPA Vlan1
Internet 192.168.1.29 0 0023.54fa.53ef ARPA Vlan1
Internet 192.168.1.252 0 001c.c06c.e017 ARPA Vlan1
Internet 192.168.0.4 170 14da.e968.d0b7 ARPA Vlan1
Internet 192.168.0.42 0 14da.e968.d032 ARPA Vlan1
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.0.129 0 0026.180a.7301 ARPA Vlan1
Internet 192.168.0.11 0 0026.1823.9d94 ARPA Vlan1
Internet 192.168.0.177 0 7071.bc07.33dc ARPA Vlan1
Internet 192.168.1.224 0 7071.bc07.3779 ARPA Vlan1
Internet 192.168.0.155 0 0023.54fa.57ef ARPA Vlan1
Internet 192.168.1.232 0 7071.bc07.36ab ARPA Vlan1
Internet 192.168.1.9 72 7071.bc07.32f7 ARPA Vlan1
Internet 192.168.1.165 0 7071.bc37.9e0b ARPA Vlan1
Internet 192.168.0.128 0 0026.1823.9a1d ARPA Vlan1
Internet 192.168.2.41 170 7071.bc51.c713 ARPA Vlan1
Internet 192.168.2.128 0 7071.bc50.0d76 ARPA Vlan1
Internet 192.168.1.4 0 7071.bc50.1471 ARPA Vlan1
Internet 192.168.1.37 0 14da.e968.cf8e ARPA Vlan1
Internet 192.168.2.68 0 001c.c06c.d6c5 ARPA Vlan1
Internet 192.168.0.174 0 7071.bc07.3375 ARPA Vlan1
Internet 192.168.1.88 18 0026.180a.720a ARPA Vlan1
Internet 192.168.0.236 0 0026.1823.9a08 ARPA Vlan1
Internet 192.168.0.142 0 0026.180a.7351 ARPA Vlan1
Internet 192.168.1.216 0 7071.bc07.32fb ARPA Vlan1
Internet 192.168.2.203 0 0026.1823.9aaa ARPA Vlan1
Internet 192.168.1.178 157 0026.188f.21ef ARPA Vlan1
Internet 192.168.2.38 0 0026.1823.9a13 ARPA Vlan1
Internet 192.168.0.26 0 7071.bc06.2677 ARPA Vlan1
Internet 192.168.3.27 0 7071.bc50.0bdf ARPA Vlan1
Internet 192.168.2.73 0 7071.bc50.0e79 ARPA Vlan1
Internet 192.168.1.231 0 7071.bc07.3304 ARPA Vlan1
Internet 192.168.2.28 0 7071.bc07.371e ARPA Vlan1
Internet 192.168.0.78 0 7071.bc07.32f4 ARPA Vlan1
Internet 192.168.2.46 0 7071.bc50.0bf3 ARPA Vlan1
Internet 192.168.1.57 0 0026.188f.210b ARPA Vlan1
Internet 192.168.2.88 0 7071.bc51.c81c ARPA Vlan1
Internet 192.168.2.13 0 14da.e968.cb6a ARPA Vlan1
Internet 192.168.1.221 0 7071.bc06.25e2 ARPA Vlan1
Internet 192.168.1.225 0 0023.54fa.59cc ARPA Vlan1
Internet 192.168.0.25 0 14da.e968.c70e ARPA Vlan1
Internet 192.168.2.201 170 7071.bc51.c89a ARPA Vlan1
Internet 192.168.1.191 0 0019.d1e4.5634 ARPA Vlan1
Internet 192.168.1.60 0 7071.bc50.140c ARPA Vlan1
Internet 192.168.3.159 0 7071.bc50.0e07 ARPA Vlan1
Internet 192.168.1.166 0 001c.c06c.d6ae ARPA Vlan1
Internet 192.168.2.195 0 0023.54fa.51c6 ARPA Vlan1
Internet 192.168.1.159 0 0023.54fa.512a ARPA Vlan1
Internet 192.168.2.106 0 0026.185c.a63c ARPA Vlan1
Internet 192.168.3.16 0 7071.bc30.b182 ARPA Vlan1
Internet 192.168.2.105 0 0026.1831.c581 ARPA Vlan1
Internet 192.168.1.39 0 0023.54fa.59fc ARPA Vlan1
Internet 192.168.0.160 0 7071.bc50.0e31 ARPA Vlan1
Internet 192.168.2.45 0 0021.7002.6a0e ARPA Vlan1
Internet 192.168.0.21 0 7071.bc50.0dff ARPA Vlan1
Internet 192.168.0.181 0 7071.bc50.0e8a ARPA Vlan1
Internet 192.168.1.186 0 0023.54fa.5a1c ARPA Vlan1
Internet 192.168.2.158 0 a41f.7253.85ce ARPA Vlan1
Protocol Address &nb
MAC flap usually occurs when there is loop in layer two switches. Please check for your STP configuration or try shutting the redundant link down.
Hope this helps!
Here's your problem.
C 192.168.0.0/20 is directly connected, Vlan1
Tied up now. Will try to come back later.
Hi,
How many hosts approximatelly are connected to this single broadcast domain?
Have you change any of the arp/mac timers on your switch?
If you are getting mac flaps it seems to me it would be a L2 problem possible bridge loop. Maybe a dumb hub /switch in the network or LACP on one one side and misconfiged on the other?
Can you trace down the source the the MAC being learned/relearned to a singe switch? Try unplugging one of the ports that is getting the mac flap and see if that resolves the issue. If not trace it back to the problem switch.
Example
sw1 ------sw2----sw3----dumb hub (no stp)
| |
|---------sw4----sw5------sw6
In this case you would see mac learned/relearned (so many per second) on all the switches in this path. On sw1 you will see mac flaps on the port facing sw2 and sw4. On sw4 you will see it on the ports facing sw5 and sw1. ETC.
On Sw1 when you remove the port facing Sw2 or Sw4 the problem should go away immediatly, but users on the dowstream would not be able to connect. So if we remove Sw2 and the problem goes away plug it back in and remove the dowstream port to sw3. Keep working your way back until you find the dumb hub and remove it.
I was thinking along the same line. What I see here is a giant flat layer II network with 700+ hosts on the same vlan. Based on the information posted, it apears the arp aging timer is set to 180 mins instead of the 4 hours default value seen on Cisco switches. What this means is that all the intra-vlan communications -which could be a lot- is done through switching as opposed to routing. This inevitably will result in tons of arping. And the fast aging of arp entries just made things a lot worse.
Of course, the occasional MAC flapping will add CPU cycles too. But they won't show up on the "Arp Input" line though.
Just my 2c
+++ copied from original post
CPU utilization for five seconds: 99%/12%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
12 111438973 18587995 5995 44.47% 43.88% 43.96% 0 ARP Input
Hi,
ITs not quite a flat later 2 network as host are on vlan 1 server, printers, phones, wifi and managment all have there own vlan. I have check the stp stuff and tthe correct device is root for the netowrk and the correct port are fowarding and blocking. I used "ip accounting mac-address" on interface vlan 1to see to top mac address that are talking and shut them down. This seems to have had some effect but was only tempory. I am sure I am getting alot of arp messages
ARP statistics:
Rcvd: 175893523 requests, 16526 replies, 1524 reverse, 0 other
Sent: 703952 requests, 8392279 replies (42155 proxy), 0 reverse
Drop due to input queue full: 4453836
Vlan1
Protocol Pkts In Chars In Pkts Out Chars Out
IP 31557843 6047919076 3098013 240366653
ARP 175531088 10531865280 8307873 498472380
Vlan10
Protocol Pkts In Chars In Pkts Out Chars Out
IP 135926 12108680 116272 10696446
ARP 15202 912124 4196 251760
Vlan20
Protocol Pkts In Chars In Pkts Out Chars Out
ARP 51 3060 2 120
Vlan30
Protocol Pkts In Chars In Pkts Out Chars Out
Other 78 4762 0 0
IP 299939 40698661 138852 16310438
Spanning Tree 1 97 0 0
ARP 226641 13598460 129423 7765380
Vlan40
Protocol Pkts In Chars In Pkts Out Chars Out
IP 305934 30987832 200110 21887686
ARP 2962 177720 2034 122040
Vlan50
Protocol Pkts In Chars In Pkts Out Chars Out
IP 119934 17060183 116516 10715825
ARP 25196 1511760 550379 33022740
Vlan60
Protocol Pkts In Chars In Pkts Out Chars Out
Other 1524 91440 0 0
IP 479094 71684264 1674193 205206465
ARP 217260 13035600 90270 5416200
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 16385
Address 6c20.5660.a900
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
but am not sure why or where its coming from. There are unmanaged switches in the network which were removed and this did not slove the issue.
also forgot to add that the arp age has not been changed and should be the default
Shrink your L2 domain. /20 is very large. Subdivide and conquer.
Don't use VLAN1 for production traffic. Pick any other VLAN, preferably several and split this up (as mentioned above).
Is your STP topology reasonable and well-designed? Is it even designed at all, or pure luck? Is every switch running RSTP?
Hi,
I have been recommending uses vlans per area so give subnets no bigger then a /24 for nearly 11 months. The stp topology is well-designed and I am sure this is not causing the issues. As a test all the redudency was taken off so there are no loops in the network and this issues still continued with no signs of improvement. The basic network has 2 core switches. The first switch( the one with the hi cpu) is set a root for all the vlan using a vlaue and not the spanning root command. Also all the interface have been check accross the network to see if everything is fowarding or blocking correctly and this has been the case. The network is running rapid spanning tree with 1 instance per vlan (rapid-pvst+). All trunk links run udld and all access ports are defined a Edge P2p ports and have bpdu guard. Every switch had config redone and was tested and had been working fine for months.
my personal view is this is a faulty device and/or a infected pc. I have made a plan to set up some vlans in the coming days and will see if this has an effect on the network.
FYI, the spanning root command just updates the config with "a value".
What do you see with 'sh spann | i gy changes'?
What STP features have you deployed to further stabilize the network?
What debugs have you left running into your logging buffer in order to be certain it's not an STP problem?
Wait, so MAC flaps continued when you removed all redundancy?
I've seen MAC flaps with Q-in-Q. In case you have that configured.
I saw the word "wifi" mentioned earlier. Mac flapping is pretty common on networks with wireless clients. It happens when mobile users are roaming from AP to AP. That in itself should not be a problem.
Hi,
Yes there are still mac flaps with no redundency at all and no unmanaged switch/hubs. All the mac flaps are on vlan 1. There is not q-in-q config at all. Wifi could have been a possible issue but I moved all wifi connection (controlers and AP's) to its own vlan and there are no mac flaps on that vlan. Going thought diagnostics I saw something that I could not explane. I saw a mac flap bewteen a port that was a trunk link and a port that was connected to a pc. Thinking this was strange I checked the port connected to the pc........
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 001c.c06c.d620 DYNAMIC Gi1/1/3
1 001c.c06c.d694 DYNAMIC Gi1/1/3
1 001c.c06c.d6ac DYNAMIC Gi1/1/3
1 001c.c06c.d6e3 DYNAMIC Gi1/1/3
1 001c.c06c.d78c DYNAMIC Gi1/1/3
1 001c.c06c.d7fc DYNAMIC Gi1/1/3
1 001c.c06c.d879 DYNAMIC Gi1/1/3
1 0023.54fa.512a DYNAMIC Gi1/1/3
1 0023.54fa.561b DYNAMIC Gi1/1/3
1 0024.8c63.83b3 DYNAMIC Gi1/1/3
1 0024.8c8f.94a1 DYNAMIC Gi1/1/3
1 0026.180a.7025 DYNAMIC Gi1/1/3
1 0026.180a.7045 DYNAMIC Gi1/1/3
1 0026.180a.7205 DYNAMIC Gi1/1/3
1 0026.180a.7294 DYNAMIC Gi1/1/3
1 0026.180a.72ad DYNAMIC Gi1/1/3
1 0026.180a.72bd DYNAMIC Gi1/1/3
1 0026.180a.7328 DYNAMIC Gi1/1/3
1 0026.1823.99e4 DYNAMIC Gi1/1/3
1 0026.1823.99fa DYNAMIC Gi1/1/3
1 0026.1823.9a09 DYNAMIC Gi1/1/3
1 0026.1823.9a8e DYNAMIC Gi1/1/3
1 0026.1823.9a94 DYNAMIC Gi1/1/3
1 0026.1823.9f66 DYNAMIC Gi1/1/3
1 0026.1823.9f6e DYNAMIC Gi1/1/3
1 0026.185c.a646 DYNAMIC Gi1/1/3
1 0026.185c.a9c9 DYNAMIC Gi1/1/3
1 0026.186e.ad4a DYNAMIC Gi1/1/3
1 0026.188f.242a DYNAMIC Gi1/1/3
1 14da.e968.c829 DYNAMIC Gi1/1/3
1 14da.e968.ca47 DYNAMIC Gi1/1/3
1 14da.e968.cfb4 DYNAMIC Gi1/1/3
1 60a4.4c64.8b1d DYNAMIC Gi1/1/3
1 7071.bc06.24d1 DYNAMIC Gi1/1/3
1 7071.bc06.2525 DYNAMIC Gi1/1/3
1 7071.bc06.2641 DYNAMIC Gi1/1/3
1 7071.bc07.32fb DYNAMIC Gi1/1/3
1 7071.bc07.3313 DYNAMIC Gi1/1/3
1 7071.bc07.36ad DYNAMIC Gi1/1/3
1 7071.bc07.36f7 DYNAMIC Gi1/1/3
1 7071.bc50.0cde DYNAMIC Gi1/1/3
1 7071.bc50.0f15 DYNAMIC Gi1/1/3
1 7071.bc51.c7a9 DYNAMIC Gi1/1/3
1 90e6.ba17.b2a0 DYNAMIC Gi1/1/3
1 90e6.ba17.b69b DYNAMIC Gi1/1/3
1 a41f.7286.39e8 DYNAMIC Gi1/1/3
1 a41f.7286.3d19 DYNAMIC Gi1/1/3
Total Mac Addresses for this criterion: 47
47 mac address from a PC? Virus checked and spyware checked all clean rebooted the pc and learnt 1 mac address. there are no virtual interface or and additional config on the pc. after around 10 minutes I start learning multiple mac address on that same port again.
Your arp traffic is off the chart. There might be something else going on here. I'd hook up a network sniffer and try to find out where all those arp requests are coming from. Perhaps some virus or malicious program?
Do you have wireless clients on this VLAN? MAC flaps could be a consequence of your clients roaming between APs, so sometimes it is not a problem at all, just a sign that the clients are roaming through your facility, which is normal.
Interesting. There definitely is more stuff going on here than meets the eye. Sounds like virus. But I would manually trace the cable on g1/1/3 to make sure it's not connected back to some other switch thereby forming a L2 loop. (Temporarily) kill the port may be a good idea too if that's feasible.
So only one of the core switches is having high cpu? What do arp stats on the second core switch look like? Do they match that on the first switch?
Definitely worth double-checking the cabling.
You could also use port-security on that port with a very limited number of MAC addresses.
Hi,
I am sure there is something weard going on. I have done a wireshark capture and can see different computers on the network arping for eachother althought they should not be. I have been asured ther are no viruses or programs that caused caused this althought I am not so sure.
The port 1/1/3 was set up by me so I can have contivity to the core switch as I shutdown the connections to all of ther ohter switches. It is 100% granted to be the pc with no devicedc in the middle. Multicast was enabled on the switch and when removed looked like it helped but then the rest of then network was back its the same stroy. I put a section of desktops in its own vlan and ( around 70 desktops) and it looks like it has not really helped. I am thinking to talk everything off vlan 1 and test
I also noticed other ports on access swithces that had larg numbers of mac address (100+) to ports that were connected to pc's. I will have to do alot more tshooting to find the cause.
If this is true, setup a port-security on that port to only allow your MAC Address and then set the violation mode to restrict. As soon as you do that, you will keep your connection up and running and then if all the other MACs will fail. A lot of things will start to have problems and then you will see what's going on.
Do know the manufacturers of the PCs - you OUIs from -
001cc0 - Intel
002354 - ASUSTek COMPUTER INC
00248c - As above
002618 - As above
14dae9 - As above
60a44c - As above
90e6ba - As above
7071bc - PEGATRON CORPORATION
a41f72 - DELL INC
Have you got a hub or switch on the end of this port? If not can you share you Wireshark traces? Have you captured other data other than ARP?
Do you have PCs from these manufacturers?
Could also be a misconfigured port-channel. You will definitely see mac-flaps if you have PC configured at one end, but not at other - this will not show up in a valid STP topology, as the topology is still valid (root bridges, stp priorities etc), but you'll still get a loop.
I had a similar problem on my network few years back. Do you have any GLBP/HSRP running? If so make sure the active forwarding device is the spanning tree root. You might still have looping traffic through unexpected paths(.e.g. at access layer). You need smaller subnets e.g. /24 and move everything off vlan 1 and prune vlan one off trunk links. Vlan 1 is carries control traffic and when you have spanning-tree noise or other protocol noise it just makes things worse with everything else right there.
Make sure proxy arp is disabled on router interfaces as hosts with misconfigured IP/Subnet masks can trigger extra arp noise with the router responding on behalf of the hosts "gateway". Also ensure there is no other device connected that has proxy-arp enabled-in show ip arp output look out for many MAC addresses resolving to the same IP Address.
Hi,
We are aware of the mac address and know some of the pc's its coming from.
Port security is not an option at the moment as pc get moved around and we can
have them shutdown. Also there are around 20-30 access switches so it would be
a time consuming job. This can be scheduled in but won’t be for a while.
Most of our PC are RM and they have some components from these manufactures.
We did look up the mac first thing when we saw the issue.
Miss configured port channels do cause mac flaps but in this case I check
this first and they are all fine (switch and server side). I believe the mac
flaps are the cause of desktops advertising multiple mac address to the
switches. I noticed at least 4 ports on different switches that are doing the
same thing and learn the same mac address on.
We are running HSRP and it is the root bridge that is master for HSRP. I
disabled proxy arp when I got this issue as a first step. I have managed to
reduce the overall cpu to 70-60% and arp input to 20-25% after powering down
all computers and then reloading all switches. The source may have been a
server which we are investigating. The arp is still very high for my liking and
we are continuing to see what we can do. Cisco tac have been involved and they
have not had any luck working out what is going on.
Dynamic arp was an idea but can’t do it as some of the edge switches are
2950 and its not supported. A big thanks to everyone for your grate ideas still
open to more.