CDP Mystery - Mutiple devices on same interface

Alright, this isn't study related, this is a real world headache that cropped up. But, anything cna be a learning opportunity, and I'm completely stumped.

 

I've got a switch, and I run show cdp neighbor on it. 

 

I see:


VASwitch5                    Gig 0/47              144           R S I     WS-C3560G-Gig 0/48

vamarx03                    Gig 0/45              154            S I      WS-C3560-4Fas 0/48

VASwitch6                    Gig 0/48              144           R S I     WS-C3560-4Fas 0/48

VAMARX02                    Gig 0/44              139            S I      WS-C3560G-Gig 0/48

VAMARX00                    Gig 0/48              162           R S I     WS-C3560G-Gig 0/48

VAMARX01                    Gig 0/43              176            S I      WS-C3560G-Gig 0/48

vamarx04                    Gig 0/46              161            S I      WS-C3560-4Fas 0/48


See anything funny with VASwitch6 and VAMARX00? That's right, they're on hte same interface. But they're not. How do I know? Because the switch I'm trying this command from IS switch VAMARX00. And since they're not just echoing back information (you can tell they're not - different hardware platforms on 0 and 6). 


So, can anyone give me an idea what could be happening? I won't be visiting the site for a couple weeks, but the tech on site absolutely positively guarantees me that the connection between VAMARX00 and VASwitch6 is a direct connection. No hubs, nothing fancy, not even a patch panel. 


Thanks

-Joe

Comments

  • Q-in-q tunneling setup is your best bet. CDP entries might also have more entries on a interface then one if the other is not timed out yet (i think).

  • Well...two things. 

     

    First, yes, I should have thought of an unexpired timer on the interface. Not because that's the correct answer, but I should have mentioned that I'd been working on the stupid thing for an hour trying to figure out anything that made sense*. 

     

    Two, since I'd been at it for an hour, obviously the 180 second CDP timeout had expired multilpe times. 

     

    So, not that. 

     

    As for QinQ tunneling, I'll have to check more deeply, but the guy I took this over from was painfully incompetent. So, while I suppose it's possible, I really doubt that's what it'll be. I'll check it in the morning, though. 

     

    Anyone else have any ideas? 

     

    -Joe

     

    *As part of wrestling with this, I did learn that the MAC addresses of SVIs will propogate to attached switches, but will not show up in the MAC address table of the switch holding the SVIs. However, they will show up in the ARP table of that switch...

  • Are you getting packet loss?  What is the CPU history at?  I'm going to agree with the previous answers. It's most likely either a cabling problem or some kind of tunnel misconfig. 

    Brian McGahan, CCIE #8593 (R&S/SP/Security)

    Internetwork Expert, Inc.

    On Mar 19, 2012, at 6:18 PM, "Merijeek" <[email protected]> wrote:

    Well...two things. 

     

    First, yes, I should have thought of an unexpired timer on the internet. Not because that's the correct answer, but I should have mentioned that I'd been working on the stupid thing for an hour trying to figure out anything that made sense*. 

     

    Two, since I'd been at it for an hour, obviously the 180 second CDP timeout had expired multilpe times. 

     

    So, not that. 

     

    As for QinQ tunneling, I'll have to check more deeply, but the guy I took this over from was painfully incompetent. So, while I suppose it's possible, I really doubt that's what it'll be. I'll check it in the morning, though. 

     

    Anyone else have any ideas? 

     

    -Joe




    INE - The Industry Leader in CCIE Preparation

    http://www.INE.com



    Subscription information may be found at:

    http://www.ieoc.com/forums/ForumSubscriptions.aspx
  • I'll pull some config tomorrow and post it.

     

    I'm kind of relieved that this isn't something simple enough that I really should have known the answer. I'll also see about getting the cable swapped, but that'll likely be an after-hours matter tomorrow.

     

    -Joe

  • Let's see...looks like I can't post an attachment, so I guess this one is going to be a bit long. 

    CDP output, this morning, a good 15 hours after the last one:

     

    VASwitch6                    Gig 0/48              151           R S I     WS-C3560-4Fas 0/48

     

    VAMARX00                    Gig 0/48              170           R S I     WS-C3560G-Gig 0/48


    I've removed the extraneous switches. So, just these two.

     

     

     

    VAMARX00#show run int g0/48

    Building configuration...

     

    Current configuration : 323 bytes

    !

    interface GigabitEthernet0/48

     description Switch6

     switchport trunk encapsulation dot1q

     switchport mode trunk

     speed 100

     duplex full

     srr-queue bandwidth share 10 10 60 20

     srr-queue bandwidth shape  10  0  0  0

     queue-set 2

     priority-queue out

     mls qos trust dscp

     auto qos voip trust

     spanning-tree portfast


    Yes, it is a 100Mbit uplink. That'll be getting fixed soon. Obviously, irrelevent to our current issue.



    GigabitEthernet0/48 is up, line protocol is up (connected)

      Hardware is Gigabit Ethernet, address is 0016.c8c9.1e30 (bia 0016.c8c9.1e30)

      Description: Switch6

      MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

         reliability 255/255, txload 1/255, rxload 2/255

      Encapsulation ARPA, loopback not set

      Keepalive set (10 sec)

      Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

      input flow-control is off, output flow-control is unsupported

      ARP type: ARPA, ARP Timeout 04:00:00

      Last input 00:00:01, output 00:00:00, output hang never

      Last clearing of "show interface" counters 12w6d

      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

      Queueing strategy: fifo

      Output queue: 0/40 (size/max)

      5 minute input rate 830000 bits/sec, 621 packets/sec

      5 minute output rate 608000 bits/sec, 347 packets/sec

         1797321411 packets input, 3268609881 bytes, 0 no buffer

         Received 7939566 broadcasts (0 multicast)

         0 runts, 0 giants, 0 throttles

         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

         0 watchdog, 513439 multicast, 0 pause input

         0 input packets with dribble condition detected

         1376343380 packets output, 2447892465 bytes, 0 underruns

         0 output errors, 0 collisions, 0 interface resets

         0 babbles, 0 late collision, 0 deferred

         0 lost carrier, 0 no carrier, 0 PAUSE output

         0 output buffer failures, 0 output buffers swapped out



         Transmit GigabitEthernet0/48             Receive

       2911159408 Bytes                       2941880598 Bytes

       2738223226 Unicast frames              3224580802 Unicast frames

        151507735 Multicast frames               1289699 Multicast frames

         64380471 Broadcast frames              11548990 Broadcast frames

                0 Too old frames              1701510472 Unicast bytes

                0 Deferred frames              253316160 Multicast bytes

                0 MTU exceeded frames          936138813 Broadcast bytes

                0 1 collision frames                   0 Alignment errors

                0 2 collision frames                   1 FCS errors

                0 3 collision frames                   0 Oversize frames

                0 4 collision frames                   0 Undersize frames

                0 5 collision frames                   0 Collision fragments

                0 6 collision frames

                0 7 collision frames            28482173 Minimum size frames

                0 8 collision frames          2983408775 65 to 127 byte frames

                0 9 collision frames            41365176 128 to 255 byte frames

                0 10 collision frames            6922691 256 to 511 byte frames

                0 11 collision frames          127186878 512 to 1023 byte frames

                0 12 collision frames           50053799 1024 to 1518 byte frames

                0 13 collision frames                  0 Overrun frames

                0 14 collision frames                  0 Pause frames

                0 15 collision frames

                0 Excessive collisions                 0 Symbol error frames

                0 Late collisions                      0 Invalid frames, too large

                0 VLAN discard frames                  0 Valid frames, too large

                0 Excess defer frames                  0 Invalid frames, too small

         75603930 64 byte frames                       0 Valid frames, too small

       2670491425 127 byte frames

        145992974 255 byte frames                      0 Too old frames

          7867955 511 byte frames                      0 Valid oversize frames

          4020884 1023 byte frames                     0 System FCS error frames

         50134264 1518 byte frames                     0 RxPortFifoFull drop frame

                0 Too large frames

                0 Good (1 coll) frames

                0 Good (>1 coll) frames





    VAMARX00#show int g0/48 tru


    Port        Mode         Encapsulation  Status        Native vlan

    Gi0/48      on           802.1q         trunking      1


    Port        Vlans allowed on trunk

    Gi0/48      1-4094


    Port        Vlans allowed and active in management domain

    Gi0/48      1,10,20,30,40,50,100,110,120,172,200,888,999


    Port        Vlans in spanning tree forwarding state and not pruned

    Gi0/48      1,10,20,30,40,50,100,110,120,172,200,888,999





    VAMARX00#show int g0/48 swi

    Name: Gi0/48

    Switchport: Enabled

    Administrative Mode: trunk

    Operational Mode: trunk

    Administrative Trunking Encapsulation: dot1q

    Operational Trunking Encapsulation: dot1q

    Negotiation of Trunking: On

    Access Mode VLAN: 1 (default)

    Trunking Native Mode VLAN: 1 (default)

    Administrative Native VLAN tagging: enabled

    Voice VLAN: none

    Administrative private-vlan host-association: none

    Administrative private-vlan mapping: none

    Administrative private-vlan trunk native VLAN: none

    Administrative private-vlan trunk Native VLAN tagging: enabled

    Administrative private-vlan trunk encapsulation: dot1q

    Administrative private-vlan trunk normal VLANs: none

    Administrative private-vlan trunk private VLANs: none

    Operational private-vlan: none

    Trunking VLANs Enabled: ALL

    Pruning VLANs Enabled: 2-1001

    Capture Mode Disabled

    Capture VLANs Allowed: ALL


    Protected: false

    Unknown unicast blocked: disabled

    Unknown multicast blocked: disabled

    Appliance trust: none




    The only damning thing I've noticed is that it's running a kind of old version:



    VAMARX00#show ver

    Cisco IOS Software, C3560 Software (C3560-ADVIPSERVICESK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

    Copyright (c) 1986-2006 by Cisco Systems, Inc.

    Compiled Fri 28-Jul-06 12:34 by yenanh

    Image text-base: 0x00003000, data-base: 0x012237D0


    ROM: Bootstrap program is C3560 boot loader

    BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(25r)SE1, RELEASE SOFTWARE (fc)


    VAMARX00 uptime is 37 weeks, 5 days, 22 hours, 57 minutes

    System returned to ROM by power-on

    System restarted at 09:20:10 EDT Wed Jun 29 2011

    System image file is "flash:c3560-advipservicesk9-mz.122-25.SEE2.bin"



    This product contains cryptographic features and is subject to United

    States and local country laws governing import, export, transfer and

    use. Delivery of Cisco cryptographic products does not imply

    third-party authority to import, export, distribute or use encryption.

    Importers, exporters, distributors and users are responsible for

    compliance with U.S. and local country laws. By using this product you

    agree to comply with applicable laws and regulations. If you are unable


    VAMARX00#show ver

    Cisco IOS Software, C3560 Software (C3560-ADVIPSERVICESK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

    Copyright (c) 1986-2006 by Cisco Systems, Inc.

    Compiled Fri 28-Jul-06 12:34 by yenanh

    Image text-base: 0x00003000, data-base: 0x012237D0


    ROM: Bootstrap program is C3560 boot loader

    BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(25r)SE1, RELEASE SOFTWARE (fc)


    VAMARX00 uptime is 37 weeks, 5 days, 22 hours, 57 minutes

    System returned to ROM by power-on

    System restarted at 09:20:10 EDT Wed Jun 29 2011

    System image file is "flash:c3560-advipservicesk9-mz.122-25.SEE2.bin"



    [Junk snipped]



    512K bytes of flash-simulated non-volatile configuration memory.

    Base ethernet MAC Address       : 00:16:C8:C9:1E:00

    Motherboard assembly number     : 73-9358-04

    Power supply part number        : 341-0107-01

    Motherboard serial number       : FOC10043B8U

    Power supply serial number      : FXD094601KN

    Model revision number           : C0

    Motherboard revision number     : A0

    Model number                    : WS-C3560G-48TS-E

    System serial number            : FOC1004Z5U7

    SFP Module assembly part number : 73-7757-03

    SFP Module revision Number      : A0

    SFP Module serial number        : CAT10021LK0

    Top Assembly Part Number        : 800-26347-02

    Top Assembly Revision Number    : B0

    Version ID                      : V02

    CLEI Code Number                : CNMWX00ARB

    Hardware Board Revision Number  : 0x05



    Switch   Ports  Model              SW Version              SW Image

    ------   -----  -----              ----------              ----------

    *    1   52     WS-C3560G-48TS     12.2(25)SEE2            C3560-ADVIPSERVICESK





    Finally, just for fun, a show cdp nei detail for both alleged devices.


    The one that's supposed to be there:



    -------------------------

    Device ID: VASwitch6

    Entry address(es):

      IP address: 10.4.3.246

    Platform: cisco WS-C3560-48PS,  Capabilities: Router Switch IGMP

    Interface: GigabitEthernet0/48,  Port ID (outgoing port): FastEthernet0/48

    Holdtime : 165 sec


    Version :

    Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

    Copyright (c) 1986-2006 by Cisco Systems, Inc.

    Compiled Fri 28-Jul-06 12:34 by yenanh


    advertisement version: 2

    Protocol Hello:  OUI=0x00000C, Protocol ID=0x0112; payload len=27, value=00000000FFFFFFFF010221FF0000000000000017E0058A80FF0000

    VTP Management Domain: 'XXX.com'

    Native VLAN: 1

    Duplex: full

    Management address(es):

      IP address: 10.4.3.246




    And VAMARX00 seeing itself:



    Device ID: VAMARX00

    Entry address(es):

      IP address: 10.4.1.4

    Platform: cisco WS-C3560G-48TS,  Capabilities: Router Switch IGMP

    Interface: GigabitEthernet0/48,  Port ID (outgoing port): GigabitEthernet0/48

    Holdtime : 68 sec


    Version :

    Cisco IOS Software, C3560 Software (C3560-ADVIPSERVICESK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

    Copyright (c) 1986-2006 by Cisco Systems, Inc.

    Compiled Fri 28-Jul-06 12:34 by yenanh


    advertisement version: 2

    Protocol Hello:  OUI=0x00000C, Protocol ID=0x0112; payload len=27, value=00000000FFFFFFFF010221FF0000000000000016C8C91E00FF0000

    VTP Management Domain: 'XXX.com'

    Native VLAN: 1

    Duplex: full

    Management address(es):

      IP address: 10.4.1.4




    So...going to update and get the cable switched after hours tonight. Anyone else have any great ideas?


    -Joe

     

  • One tool that I've found very helpful in troubleshooting potential remote cabling issues is the built in tdr command. Depending on your ios version, it may be service impacting. 

     

     

    # test cable-diagnostics tdr interface gigabitEthernet 0/48

    # show cable-diagnostics tdr interface gigabitEthernet 0/48

  • I'm not sure why you have the access mode portfast configured on a trunk port. It shouldn't cause any issue as it's only effective if the port is NOT trunking, but still shouldn't be there.

  • Well, I wasn't responsible for the configuration of this site (see above about the total incompetent), so it wasn't me who put that portfast in there. 

     

    Still, it shouldn't have mattered, but since there's no reason for it to be in there I did remove that command. No effect on the CDP oddness.

     

    I don't see anything on Cisco's site about the TDR command dropping the connection, but since there's about 30 phones located downstream from that switch, I can't afford to take the chance of dropping it right now, so I'll have to try it after hours. 

     

     

  • Further interestingliness. 

     

    I do the TDR, and I get: 

     


    Interface Speed Local pair Pair length        Remote pair Pair status

    --------- ----- ---------- ------------------ ----------- --------------------

    Gi0/48    100M  Pair A     1    +/- 4  meters Pair A      Normal

                    Pair B     1    +/- 4  meters Pair B      Normal

                    Pair C     2    +/- 4  meters Pair C      Short

                    Pair D     3    +/- 4  meters Pair D      Short



    However, on all three gigabit ports on this switch that are trunked to an FE port at the other end, I get the exact same result. So, I'm thinking it's a false positive, so I'm going to get those cables changed. 


    So, after doing the tdr the switch no longer sees itself in the CDP neighbor statement. For about 2 minutes - switch 6, the other device that is actually on the same port, comes back in...probably 10-15 seconds. 


    Still stumped. We'll see what happens after the cable gets changed. 

  • Can you post the config for the switchport on the far end switch (VASwitch6)? I'm curious how port Fa0/47 is configured.


  • interface FastEthernet0/48

     description Uplink to Switch0

     switchport trunk encapsulation dot1q

     switchport mode trunk

     speed 100

     duplex full

     srr-queue bandwidth share 10 10 60 20

     srr-queue bandwidth shape  10  0  0  0

     priority-queue out

     mls qos trust dscp

     auto qos voip trust

     spanning-tree portfast

    end


    And, yes, I did take out the portfast after doing this show run. 



    -Joe

  • Okay, so, I finally got to the site. 

    Since the switches were running antique software, I upgraded them to c3560-ipservicesk9-mz.122-58.SE2. 

    Change out the cable between the switches, and...

     

    VAMARX05

                     Gig 0/47          160             R S I  WS-C3560G Gig 0/48

    vamarx04

                     Gig 0/46          144              S I   WS-C3560- Fas 0/48

    vamarx06

                     Gig 0/48          149             R S I  WS-C3560- Fas 0/48

    VAMARX01

                     Gig 0/43          166              S I   WS-C3560G Gig 0/48

    VAMARX00

                     Gig 0/48          149             R S I  WS-C3560G Gig 0/48

    vamarx03

                     Gig 0/45          133              S I   WS-C3560- Fas 0/48

    VAMARX02

                     Gig 0/44          167              S I   WS-C3560G Gig 0/48


    So, short answer, no change. 


    I've got one more night after hours to give something a shot if someone has an idea. 


    -Joe

     

  • Since you are at the site now, did you check if this is a direct connection indeed nothing in between vamarx06 and vamarx00?

    Does vamarx06 has a single uplink connection to vamarx00? Is there any hub connected to vamarx06 that may potentionally connect back to vamarx00?

    I would like to see the output of the sh spann root from both vamarx00 and vamarx06, is this possible?

    Thanks!

  • Definitely a direct connection - I changed the cable myself, so I know it's a direct connection. 

     

    There's only a single uplink. I can't guarantee that there isn't a hub looping between 0 and 6, but when I look at the MAC table on both switches I only see a couple ports with multiple MACs on them. Even then, worst one has 5 MACs on it. One of the things I've got on my to do list is to do a hub hunt and get those scheduled to be yanked. 

    The show span root is:

     

     

    VAMARX00#

                                            Root    Hello Max Fwd

    Vlan                   Root ID          Cost    Time  Age Dly  Root Port

    ---------------- -------------------- --------- ----- --- ---  ------------

    VLAN0001         24577 0016.c8c9.1e00         0    2   20  15

    VLAN0010         24586 0016.c8c9.1e00         0    2   20  15

    VLAN0020         24596 0016.c8c9.1e00         0    2   20  15

    VLAN0030         24606 0016.c8c9.1e00         0    2   20  15

    VLAN0040         24616 0016.c8c9.1e00         0    2   20  15

    VLAN0050         24626 0016.c8c9.1e00         0    2   20  15

    VLAN0100         24676 0016.c8c9.1e00         0    2   20  15

    VLAN0110         24686 0016.c8c9.1e00         0    2   20  15

    VLAN0120         24696 0016.c8c9.1e00         0    2   20  15

    VLAN0172         24748 0016.c8c9.1e00         0    2   20  15

    VLAN0200         32968 0016.47f5.5c00         4    2   20  15  Gi0/44

    VLAN0888         33656 0016.47f5.5c00         4    2   20  15  Gi0/44

    VLAN0999         33767 0016.47f5.5c00         4    2   20  15  Gi0/44

     

    VAMARX06#




                                            Root    Hello Max Fwd

    Vlan                   Root ID          Cost    Time  Age Dly  Root Port

    ---------------- -------------------- --------- ----- --- ---  ------------

    VLAN0001         24577 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0010         24586 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0020         24596 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0030         24606 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0040         24616 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0050         24626 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0100         24676 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0110         24686 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0120         24696 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0172         24748 0016.c8c9.1e00        19    2   20  15  Fa0/48

    VLAN0200         32968 0016.47f5.5c00        23    2   20  15  Fa0/48

    VLAN0888         33656 0016.47f5.5c00        23    2   20  15  Fa0/48

    VLAN0999         33767 0016.47f5.5c00        23    2   20  15  Fa0/48




    Not really sure why the VAMARX00 isn't the root on all of them, but honestly, I'm pretty sure that VLANs 200, 888, and 999 don't actually have any ports assinged to them anywhere on the network.

     

Sign In or Register to comment.