Lab 4 - IPv4 Unicast/MPLS blackhole

I'm running into a very strange issue on Lab 4.  When trying to ping from a R4's loopback in AS 2000 to XR2's loopback in AS 1000 traffic flows correctly (with labels) but then dies at XR1 in AS1000.

I traced hop by hop following how the prefix was learned and looked at the MPLS labels and forwarding for each hop.

For example when I ussue the following from R4:

ping 10.0.0.20 source lo0 repeat 1000000 timeout 0

 

The taffic goes R4->R3->R5->XR2->XR1 (and dies)   Never makes it it back to the next hop, which XR1 says is R3.

R4#show ip route 10.0.0.20
Routing entry for 10.0.0.20/32
Known via "ospf 1", distance 110, metric 1
Tag 1000, type extern 2, forward metric 1
Last update from 10.0.34.3 on FastEthernet0/0.34, 02:13:46 ago
Routing Descriptor Blocks:
* 10.0.34.3, from 10.0.0.3, 02:13:46 ago, via FastEthernet0/0.34
Route metric is 1, traffic share count is 1
Route tag 1000

R4#show mpls forwarding-table | inc 10.0.0.20
25 26 10.0.0.20/32 0 Fa0/0.34 10.0.34.3


R3#show ip route 10.0.0.20
Routing entry for 10.0.0.20/32
Known via "bgp 2000", distance 20, metric 10
Tag 1000, type external
Redistributing via ospf 1
Advertised by ospf 1 subnets route-map BGP_TO_IGP_LABELS
Last update from 10.0.35.5 01:30:23 ago
Routing Descriptor Blocks:
* 10.0.35.5, from 10.0.35.5, 01:30:23 ago
Route metric is 10, traffic share count is 1
AS Hops 1
Route tag 1000
MPLS label: 23

R3#show mpls forwarding-table | inc 10.0.0.20
26 23 10.0.0.20/32 133610814 Fa0/0.35 10.0.35.5


R5#show ip route 10.0.0.20
Routing entry for 10.0.0.20/32
Known via "isis", distance 115, metric 10, type level-2
Redistributing via isis 1
Advertised by bgp 1000
Last update from 10.0.205.20 on FastEthernet0/0.205, 02:16:09 ago
Routing Descriptor Blocks:
* 10.0.205.20, from 10.0.0.20, 02:16:09 ago, via FastEthernet0/0.205
Route metric is 10, traffic share count is 1

R5#show mpls forwarding-table | inc 10.0.0.20
23 Pop Label 10.0.0.20/32 176392451 Fa0/0.205 10.0.205.20


We make it to XR2, now looking for packets forwarding back to R4:

RP/0/3/CPU0:XR2#show ip route 10.0.0.4
Tue Feb 29 03:12:13.496 UTC

Routing entry for 10.0.0.4/32
Known via "isis 1", distance 115, metric 10, type level-2
Installed Feb 29 03:04:44.469 for 00:07:29
Routing Descriptor Blocks
10.19.20.19, from 10.0.0.19, via POS0/7/0/0.1920
Route metric is 10
No advertising protos.

RP/0/3/CPU0:XR2#show mpls forwarding | inc 10.0.0.4
Tue Feb 29 03:12:22.712 UTC
16008 16025 10.0.0.4/32 PO0/7/0/0.1920 10.19.20.19 2960078



And here is where we arrive and die at XR1:

RP/0/0/CPU0:XR1#show ip bgp 10.0.0.4
Tue Feb 29 03:07:37.494 UTC
BGP routing table entry for 10.0.0.4/32
Versions:
Process bRIB/RIB SendTblVer
Speaker 26 26
Local Label: 16025
Last Modified: Feb 29 01:42:08.540 for 01:25:29
Paths: (1 available, best #1)
Advertised to peers (in unique update groups):
10.0.0.2
Path #1: Received by speaker 0
2000
10.0.193.3 from 10.0.193.3 (10.0.0.3)
Received Label 16
Origin IGP, metric 2, localpref 120, valid, external, best
Community: 1000:120


RP/0/0/CPU0:XR1#show mpls forwarding | inc 10.0.0.4
16025 16 10.0.0.4/32 Gi0/1/0/0.193 10.0.193.3 6122

 

Packets never go out Gig0/1/0/0.193. (as seen by the small packets forwarded)  I can also watch the POS interface on XR1 facing XR2 and I can see the packets coming in at a high rate.  The entire MPLS forwarding table from XR1:

 

RP/0/0/CPU0:XR1#show mpls forwarding
Tue Feb 29 02:48:43.181 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
16012 Pop PW(10.0.0.2:100) Gi0/1/0/1.100 point2point 98600
16015 Pop PW(10.0.0.2:200) Gi0/1/0/1.200 point2point 98532
16016 Pop 10.0.193.3/32 Gi0/1/0/0.193 10.0.193.3 14097
16017 Pop 10.0.0.20/32 PO0/6/0/0.1920 10.19.20.20 6215
16018 Pop 10.0.205.0/24 Gi0/1/0/0.195 10.0.195.5 0
Pop 10.0.205.0/24 PO0/6/0/0.1920 10.19.20.20 0
16019 Pop 10.0.220.0/24 PO0/6/0/0.1920 10.19.20.20 0
16020 16 10.0.0.2/32 Gi0/1/0/0.195 10.0.195.5 19691
16004 10.0.0.2/32 PO0/6/0/0.1920 10.19.20.20 0
16021 Pop 10.0.0.5/32 Gi0/1/0/0.195 10.0.195.5 3392
16022 Pop 10.0.25.0/24 Gi0/1/0/0.195 10.0.195.5 0
16023 Aggregate FOO: Per-VRF Aggr[V]
FOO 4680
16024 Aggregate FOO: Per-VRF Aggr[V]
FOO 0
16025 16 10.0.0.4/32 Gi0/1/0/0.193 10.0.193.3 6122
16026 Pop 10.0.0.3/32 Gi0/1/0/0.193 10.0.193.3 5568
16027 16010 10.0.0.1/32 PO0/6/0/0.1920 10.19.20.20 0

 

 

The very strange thing is that if I shutdown the POS links between XR1 and XR2 *ALL* of the traffic flows properly around the rest of the networks.  All VPNv4 routes, VRF loopbacks, in all AS's work fine.  Everything seems to be related to the hop between XR1 and XR2, but nothing is making sense.

-v10d

Comments

  • And for those interested, here was the full MPLS forwarding table on XR2:

     

    RP/0/3/CPU0:XR2#show mpls forwarding
    Tue Feb 29 03:12:08.254 UTC
    Local Outgoing Prefix Outgoing Next Hop Bytes
    Label Label or ID Interface Switched
    ------ ----------- ------------------ ------------ --------------- ------------
    16000 Pop 172.16.207.7/32[V] Gi0/4/0/0.207 172.16.207.7 812758
    16001 Pop 172.16.208.8/32[V] Gi0/4/0/0.208 172.16.208.8 801736
    16002 Pop 10.0.0.19/32 PO0/7/0/0.1920 10.19.20.19 2627
    16003 Pop 10.0.195.0/24 Gi0/4/0/0.205 10.0.205.5 520
    Pop 10.0.195.0/24 PO0/7/0/0.1920 10.19.20.19 0
    16004 Pop 10.0.0.2/32 Gi0/4/0/0.220 10.0.220.2 49010
    16005 Pop 10.0.25.0/24 Gi0/4/0/0.220 10.0.220.2 520
    16006 Pop 10.0.0.5/32 Gi0/4/0/0.205 10.0.205.5 32081
    16007 20 0 Gi0/4/0/0.205 10.0.205.5 22594
    16008 16025 10.0.0.4/32 PO0/7/0/0.1920 10.19.20.19 2959842
    16009 16026 10.0.0.3/32 PO0/7/0/0.1920 10.19.20.19 0
    16010 26 10.0.0.1/32 Gi0/4/0/0.220 10.0.220.2 25339
    16011 24 172.16.0.2/32[V] 10.0.0.2 2704
    16012 21 172.16.0.5/32[V] 10.0.0.5 520
    16013 Pop 172.16.0.7/32[V] Gi0/4/0/0.207 172.16.207.7 520
    16014 17 172.16.0.8/32[V] Gi0/4/0/0.207 172.16.207.7 520
    16015 16023 172.16.0.19/32[V] 10.0.0.19 0
    16016 Aggregate FOO: Per-VRF Aggr[V]
    FOO 2080
    16017 Unlabelled 2001:172:16::7/128[V]
     

     

  • Not sure about the lab (doing this weekend), but are the paths not congruent from R4 to XR2, and back?

    Cheers,

    AB.

  • Not sure about the lab (doing this weekend), but are the paths not congruent from R4 to XR2, and back?

     

    No, they are not.  From R4 to XR2 it goes:  R4->R3->R5->XR2

    From XR2 back to R4:  XR2->XR1->(dies)->R3(next hop per LFIB)

    -v10d

     

  • That is what I was referring to - why is the return path via XR1, shouldn't it be R5?

    AB.

  • BGP route policy setting local-preference to 120... making XR1 the better path.

    Assymetrical is fine so long as you have next-hop's and labels.  (which I seem to have).

    Labbing it up now and I'm still stumped. 

    When I shutdown POS0/6/0/0 on XR1 everything is just fine again.

    :(

    -v10d

     

  • Labbed this up again this evening.  Apparently the problem is between XR1 and R3.  BGP label exchange... for whatever it's woth it's just not working.

    If I do an MPLS trace form XR to R4 in AS2000 it just ends stating "no label".  But the prefix is learned via BGP ipv4 labeled-unicast.  I get a label in BGP and it's installed in the LFIB!  Everything looks good on both sides on R3 and XR1.  Driving me nuts!

     

    RP/0/0/CPU0:XR1# traceroute mpls ipv4 10.0.0.4/32 source 10.0.0.19
    Fri Oct 11 01:55:42.153 UTC

    Tracing MPLS Label Switched Path to 10.0.0.4/32, timeout is 2 seconds

    Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

    Type escape sequence to abort.

    0 10.0.0.19 MRU 0 [No Label]
    Q 1 *


    RP/0/0/CPU0:XR1#show ip route 10.0.0.4
    Fri Oct 11 02:02:34.475 UTC

    Routing entry for 10.0.0.4/32
    Known via "bgp 1000", distance 20, metric 2, labeled unicast (3107)
    Tag 2000, type external
    Installed Oct 11 01:31:46.255 for 00:30:48
    Routing Descriptor Blocks
    10.0.193.3, from 10.0.193.3
    Route metric is 2
    No advertising protos.


    RP/0/0/CPU0:XR1#show ip bgp 10.0.0.4
    Fri Oct 11 01:57:47.042 UTC
    BGP routing table entry for 10.0.0.4/32
    Versions:
    Process bRIB/RIB SendTblVer
    Speaker 36 36
    Local Label: 16011
    Last Modified: Oct 11 01:31:46.554 for 00:26:00
    Paths: (1 available, best #1)
    Advertised to peers (in unique update groups):
    10.0.0.2
    Path #1: Received by speaker 0
    2000
    10.0.193.3 from 10.0.193.3 (10.0.0.3)
    Received Label 16
    Origin IGP, metric 2, localpref 120, valid, external, best
    Community: 1000:120


    RP/0/0/CPU0:XR1#show bgp labels
    Fri Oct 11 01:58:16.522 UTC
    BGP router identifier 10.0.0.19, local AS number 1000
    BGP generic scan interval 60 secs
    BGP table state: Active
    Table ID: 0xe0000000
    BGP main routing table version 39
    BGP scan interval 60 secs

    Status codes: s suppressed, d damped, h history, * valid, > best
    i - internal, r RIB-failure, S stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
    Network Next Hop Rcvd Label Local Label
    *> 10.0.0.3/32 10.0.193.3 3 16012
    *> 10.0.0.4/32 10.0.193.3 16 16011


    RP/0/0/CPU0:XR1#show mpls forwarding prefix 10.0.0.4/32
    Fri Oct 11 01:56:51.859 UTC
    Local Outgoing Prefix Outgoing Next Hop Bytes
    Label Label or ID Interface Switched
    ------ ----------- ------------------ ------------ --------------- ------------
    16011 16 10.0.0.4/32 Gi0/1/0/0.193 10.0.193.3 2575

     

     

     

  • What about "show cef 10.0.0.4 detail" on XR1? I am stumped too.

    AB.

  • What about "show cef 10.0.0.4 detail" on XR1? I am stumped too.

    I'll grab that when I go back at it again tomorrow morning.  And before anyone asks, yes, the /32 static route to R3's next-hop is there.  ;)

    -v10d

  • And before anyone asks, yes, the /32 static route to R3's next-hop is there.  ;)

    -v10d

    It is obvious from the output. :)

    AB.

  • New sprack this morning labed up, same issue.  More troubleshooting... here is the same info along with the CEF enteries for both R3 (next hop) and R4 the destination:

     

    RP/0/0/CPU0:XR1#show cef 10.0.0.3/32 detail
    Fri Oct 11 13:26:12.033 UTC
    10.0.0.3/32, version 2, internal 0x40040001 (ptr 0x9cd5cd48) [1], 0x0 (0x9cd2a7a0), 0x4000 (0x9d522398)
    Updated Oct 11 13:09:47.446
    Prefix Len 32, traffic index 0, precedence routine (0)
    gateway array (0x9cbff0c4) reference count 3, flags 0xf00, source rib (3),
    [3 type 4 flags 0x101101 (0x9d554578) ext 0x0 (0x0)]
    LW-LDI[type=1, refc=1, ptr=0x9cd2a7a0, sh-ldi=0x9d554578]
    via 10.0.193.3, 3 dependencies, recursive [flags 0x20]
    next hop 10.0.193.3 via 10.0.193.3/32
    local label 16012
    next hop 10.0.193.3 Gi0/1/0/0.193 labels imposed {ImplNull ImplNull}


    Load distribution: 0 (refcount 3)

    Hash OK Interface Address
    0 Y Unknown 10.0.193.3


    RP/0/0/CPU0:XR1#show cef 10.0.0.4/32 detail
    Fri Oct 11 13:26:19.982 UTC
    10.0.0.4/32, version 2, internal 0x40040001 (ptr 0x9cd5c4f8) [1], 0x0 (0x9cd2a41c), 0x4100 (0x9d522370)
    Updated Oct 11 13:09:47.446
    Prefix Len 32, traffic index 0, precedence routine (0)
    gateway array (0x9cbfedd0) reference count 3, flags 0xf00, source rib (3),
    [2 type 5 flags 0x101101 (0x9d554488) ext 0x0 (0x0)]
    LW-LDI[type=5, refc=3, ptr=0x9cd2a41c, sh-ldi=0x9d554488]
    via 10.0.193.3, 5 dependencies, recursive [flags 0x20]
    next hop 10.0.193.3 via 16002/0/21
    local label 16013
    next hop 10.0.193.3 Gi0/1/0/0.193 labels imposed {ImplNull 20}


    Load distribution: 0 (refcount 2)

    Hash OK Interface Address
    0 Y Unknown 16002/0


    RP/0/0/CPU0:XR1#show bgp labels
    Fri Oct 11 13:26:44.253 UTC
    BGP router identifier 10.0.0.19, local AS number 1000
    BGP generic scan interval 60 secs
    BGP table state: Active
    Table ID: 0xe0000000
    BGP main routing table version 10
    BGP scan interval 60 secs

    Status codes: s suppressed, d damped, h history, * valid, > best
    i - internal, r RIB-failure, S stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
    Network Next Hop Rcvd Label Local Label
    * 10.0.0.1/32 10.0.193.3 16 nolabel
    *> 10.0.0.2/32 10.0.195.5 nolabel 16006
    * i 10.0.0.2 nolabel 16006
    *> 10.0.0.3/32 10.0.193.3 3 16012
    *> 10.0.0.4/32 10.0.193.3 20 16013
    *> 10.0.0.5/32 10.0.195.5 nolabel 16007
    * i 10.0.0.5 nolabel 16007
    *> 10.0.0.19/32 0.0.0.0 nolabel 3
    *> 10.0.0.20/32 10.19.20.20 nolabel 16003
    * i 10.0.0.20 nolabel 16003

    Processed 7 prefixes, 10 paths


    RP/0/0/CPU0:XR1#show bgp ipv4 unicast 10.0.0.4/32
    Fri Oct 11 13:27:09.008 UTC
    BGP routing table entry for 10.0.0.4/32
    Versions:
    Process bRIB/RIB SendTblVer
    Speaker 6 6
    Local Label: 16013
    Last Modified: Oct 11 13:09:47.553 for 00:17:21
    Paths: (1 available, best #1)
    Advertised to peers (in unique update groups):
    10.0.0.2
    Path #1: Received by speaker 0
    2000
    10.0.193.3 from 10.0.193.3 (10.0.0.3)
    Received Label 20
    Origin IGP, metric 2, localpref 120, valid, external, best
    Community: 1000:120

     

  • I'm at my wits end...  In an attempt to find out what's going on between R3 and XR1 I shut down links from R3 to R5 and from XR1 to XR2.  Just to isolate traffic to the path between R3 and XR1 for the InterAS LSP's...

    Now what is strange from R1, R3 and R4 in AS2000 I can get an LSP FROM their loopbacks to XR1... ie:

     

    R4#ping mpls ipv4 10.0.0.19/32 source 10.0.0.4
    Sending 5, 100-byte MPLS Echos to 10.0.0.19/32,
    timeout is 2 seconds, send interval is 0 msec:

    Type escape sequence to abort.
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

     

    However, I cannot FROM XR1 to any router in AS2000... same result as before:

    RP/0/0/CPU0:XR1#ping mpls ipv4 10.0.0.4/32 source 10.0.0.19
    Fri Oct 11 14:59:03.620 UTC

    Sending 5, 100-byte MPLS Echos to 10.0.0.4/32,
    timeout is 2 seconds, send interval is 0 msec:

    Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
    Type escape sequence to abort.

    QQQQQ
    Success rate is 0 percent (0/5)

     

    Even though I have a good CEF entry and entry in LFIB, XR1 just simply won't forward:

     

    RP/0/0/CPU0:XR1#show cef ipv4 10.0.0.4/32 detail
    Fri Oct 11 14:59:52.943 UTC
    10.0.0.4/32, version 9, internal 0x40040001 (ptr 0x9cd5cbf8) [1], 0x0 (0x9cd2a830), 0x100 (0x9d522320)
    Updated Oct 11 14:19:15.921
    Prefix Len 32, traffic index 0, precedence routine (0)
    gateway array (0x9cbff4fc) reference count 6, flags 0xf00, source rib (3),
    [3 type 5 flags 0x101101 (0x9d55471c) ext 0x0 (0x0)]
    LW-LDI[type=5, refc=3, ptr=0x9cd2a830, sh-ldi=0x9d55471c]
    via 10.0.193.3, 5 dependencies, recursive [flags 0x20]
    next hop 10.0.193.3 via 16014/0/21
    local label 16013
    next hop 10.0.193.3 Gi0/1/0/0.193 labels imposed {ImplNull 20}


    Load distribution: 0 (refcount 3)

    Hash OK Interface Address
    0 Y Unknown 16014/0
    RP/0/0/CPU0:XR1#show mpls forwarding | inc 10.0.0.4
    Fri Oct 11 15:00:14.430 UTC
    16013 20 10.0.0.4/32 Gi0/1/0/0.193 10.0.193.3 19451

     

     

  • Even though I have a good CEF entry and entry in LFIB, XR1 just simply won't forward:

     

    RP/0/0/CPU0:XR1#show cef ipv4 10.0.0.4/32 detail
    Fri Oct 11 14:59:52.943 UTC
    10.0.0.4/32, version 9, internal 0x40040001 (ptr 0x9cd5cbf8) [1], 0x0 (0x9cd2a830), 0x100 (0x9d522320)
    Updated Oct 11 14:19:15.921
    Prefix Len 32, traffic index 0, precedence routine (0)
    gateway array (0x9cbff4fc) reference count 6, flags 0xf00, source rib (3),
    [3 type 5 flags 0x101101 (0x9d55471c) ext 0x0 (0x0)]
    LW-LDI[type=5, refc=3, ptr=0x9cd2a830, sh-ldi=0x9d55471c]
    via 10.0.193.3, 5 dependencies, recursive [flags 0x20]
    next hop 10.0.193.3 via 16014/0/21
    local label 16013
    next hop 10.0.193.3 Gi0/1/0/0.193 labels imposed {ImplNull 20}


    Load distribution: 0 (refcount 3)

    Hash OK Interface Address
    0 Y Unknown 16014/0
    RP/0/0/CPU0:XR1#show mpls forwarding | inc 10.0.0.4
    Fri Oct 11 15:00:14.430 UTC
    16013 20 10.0.0.4/32 Gi0/1/0/0.193 10.0.193.3 19451

     

     

    It may very well just be my Friday night brain messing with me, but doesn't the show cef command indicate that XR1 imposes an ImplicitNull label on top of label 20, which according to the LFIB is the correct label?

    /Mark

  • It may very well just be my Friday night brain messing with me, but doesn't the show cef command indicate that XR1 imposes an ImplicitNull label on top of label 20, which according to the LFIB is the correct label?

     

    So it says, I did notice that.  Not sure but I believe the ImpNull label would be "popped" before going out to R3 leaving the Label 20 when forwarded to R3.  Seems awkward to me and no idea why it's doing it or how to fix it.  I can ping LSP's from AS2000 to routers on AS1000, but not the other way around when it comes to going through the link between R3 and XR1.  I isolated everything down to just XR1, and no matter what it just comes up with "no label" on an LSP trace.:


    RP/0/0/CPU0:XR1#traceroute mpls ipv4 10.0.0.4/32 source 10.0.0.19 verbose
    Type escape sequence to abort.

      0 10.0.0.19 0.0.0.0 MRU 0 [No Label]

    Q 1 *

     

    No idea why.  Have a good BGP learned prefix, BGP label and good entry in the LFIB for label and next hop address.

  • I have hit the exact same problem that you have but I know what the problem is.

    In Task 4.1 - Configure IPv4 Unicast EBGP peerings between R4 & XR2. 

    This causes R4 and XR2 to advertise their Loopbacks to each other. Before this task, R4 learns about XR2's Loopback via R3, and XR2 learns about R4's Loopback via R5. XR2 does not complain since it's learning R4's Loopback via R5 with a higher local-preference due to configuration task 2.9. However, R4 learns XR2's Loopback from R3 and XR2 with the same local-preference (100) because there is no configuration task to change it, and hence ends up choosing the one from XR2 causing the eBGP session to constantly flap.


    Their seems a typo - instead of IPv4 eBGP peering, it should be VPNv4 eBGP peering between R4 and XR2.


    Cheers,

    AB.


  • I actually caught that, and I don't think that's what the problem is on my end.  I think what I"m running into is a bug when testing with mpls oam on XR.

    Whenever I do an MPLS trace or ping on XR through a LSP paths that were setup using LDP it seems to work.  Whenever I traceroute MPLS or ping MPLS over a link that the has BGP labeled-unicast, it fails to find the label for the next hop.

    I started Lab4 over, skipping some of the BGP route policy stuff just to leave things in place for now.  I then setup VPNv4 between both AS's and ran into the same problem again.

    I can do a traceroute from XR now and see the MPLS labels in the path; if I use traceroute MPLS or ping MPLS Im find in AS1000, but the second I go over the path between XR1 and R3, it just dies.  (normal pings and traceroutes work)

    -v10d

  • I see the similar thing with Inter-AS Option-C in ASR9K running code 4.3.0.

    ASBR2 advertises via eBGP IPv4 Prefix for the PE (say PE2) within its ASN with Label to ASBR1. ASBR1 generates a local label for PE2 and advertises correctly to P router via LDP. P router generates a local label and advertises a label correctly to PE1 via LDP. But PE1 decides to "PoP" for PE2.

    So, instead of redistributing PE2 prefix at ASBR1, I advertised PE2 prefix to PE1 via IBGP with Label and still PE1 decides to "PoP". 

    Checking the CEF, it shows only 1 label in the stack for PE2 at PE1 router - this is completely wrong. PE1 should put 2 labels to reach PE2 - a top label to reach ASBR1, and a second label for PE2.

    This same behavior is seen on PE2 as well for PE1 prefix.

    I have done reloads, removed BGP and configured it back, clear LDP, BGP...etc..

    Cheers,

    AB.

  • The Labels are populated correctly if I remove the eBGP Multi-hop VPNv4 neighbor configuration from PE1 and PE2. Weird !!!!

    AB.

  • I did Lab4 over from start thinking that maybe I just screwed something up somewhere along the line, yet I couldn't find it.  As I said, I got everything up and running in both AS's and just skipped the lengthy BGP policies I already had.

    When I normal traceroutes I can see good LSP's between loopbacks from any PE router to any other PE router.

    The problem I run into is using "ping mpls" or "traceroute mpls" when one of the hops is on XR where there is a labeled-unicast peering.  I'm not sure what happens but XR cannot find the label for the next-hop when using these commands.  (as stated, normal traceroutes (which show labels) works fine). 

    I have to guess it's just a bug to be aware of.  I could not reproduct this problem when the entire path used LDP.

    -v10d

     

  • The problem I run into is using "ping mpls" or "traceroute mpls" when one of the hops is on XR where there is a labeled-unicast peering.  I'm not sure what happens but XR cannot find the label for the next-hop when using these commands.  (as stated, normal traceroutes (which show labels) works fine). 

     

    You need to enable OAM on the XR boxes for this to work. Just do "mpls oam" and commit, try again =) 

     

  • You need to enable OAM on the XR boxes for this to work. Just do "mpls oam" and commit, try again =)

     

    MPLS oam is enabled and configured.  If it's not, it doesn't even allow you to execute the commands and tells you to enable it.

    From my experience and troubleshooting, MPLS OAM traces/pings on LDP adjacent paths works in XR.  If they are BGP labeled-unicast, they do not work. (at least in 3.9.1).   If someone else tests and finds otherwise, please let me know.

    -v10d

  • Hi,

    I have exactly the same issue, but I do not think the problem is between  XR1 and R3. In fact, to
    simplify the topology, I shutdown the link 
    between R5 and R3, so the only inter-as connection is R3-XR1.

    After that, if try ping to R1 loopback from XR2 following the
    path XR2 -> XR1 -> R3 -> R1 and it fail. But if I shutdown the
    Pos0/7/0/0.1920 interface, it success. LDP and IGP neighbors were fine between XR1
    and XR2.

    RP/0/3/CPU0:XR2#ping 10.0.0.1 source lo0    
    Mon Nov 11 22:18:02.928 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
    .....
    Success rate is 0 percent (0/5)
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#conf t
    Mon Nov 11 22:18:17.046 UTC
    RP/0/3/CPU0:XR2(config)#inter POS0/7/0/0.1920
    RP/0/3/CPU0:XR2(config-subif)#shut
    RP/0/3/CPU0:XR2(config-subif)#commit
    RP/0/3/CPU0:Nov 11 22:18:28.654 : config[65691]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user 'xr2'. Use 'show configuration commit changes 1000002626' to view the changes.
    RP/0/3/CPU0:XR2(config-subif)#end
    RP/0/3/CPU0:Nov 11 22:18:30.284 : config[65691]: %MGBL-SYS-5-CONFIG_I : Configured from console by xr2
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#RP/0/3/CPU0:Nov 11 22:18:30.912 : bgp[139]: %ROUTING-BGP-5-ADJCHANGE : neighbor 10.0.0.4 Up (VRF: default)

    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping 10.0.0.1 source lo0
    Mon Nov 11 22:18:37.217 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#trace 10.0.0.1 source lo0
    Mon Nov 11 22:18:44.656 UTC
    % Bad hostname or protocol not running
    RP/0/3/CPU0:XR2#trace 10.0.0.1 source 10.0.0.20
    Mon Nov 11 22:18:55.262 UTC

    Type escape sequence to abort.
    Tracing the route to 10.0.0.1

     1  10.0.205.5 [MPLS: Label 23 Exp 0] 6 msec  4 msec  2 msec
     2  10.0.195.19 [MPLS: Label 16033 Exp 0] 4 msec  2 msec  4 msec
     3  10.0.193.3 [MPLS: Label 17 Exp 0] 2 msec  2 msec  2 msec
     4  10.0.13.1 2 msec  *  2 msec
    RP/0/3/CPU0:XR2#

    RP/0/3/CPU0:XR2#ping vrf FOO 172.16.0.1 source lo1
    Mon Nov 11 22:27:07.193 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/3 ms
    RP/0/3/CPU0:XR2#



    There might be an issue with POS sub-interface , but not sure.
    Also, ping and trace route work just fine from R5 to any other PE when the
    traffic pass trough XR1-R3 link.

    If anything find the cause, please share it.

    Thanks,

    David.

     


  • I isolated the issue to the POS subinterfaces between XR1
    and XR2. If you change the configuration to avoid frame-relay subinterface, the issue
    does not happen:


    interface POS0/7/0/0
     ipv4 address 10.19.20.20 255.255.255.0
     ipv6 address 2001:10:19:20::20/64

    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping mpls ipv4  10.0.0.1/32 source 10.0.0.20
    Mon Nov 11 22:40:11.166 UTC

    Sending 5, 100-byte MPLS Echos to 10.0.0.1/32,
          timeout is 2 seconds, send interval is 0 msec:

    Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
      'L' - labeled output interface, 'B' - unlabeled output interface,
      'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
      'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
      'P' - no rx intf label prot, 'p' - premature termination of LSP,
      'R' - transit router, 'I' - unknown upstream index,
      'X' - unknown return code, 'x' - return code 0

    Type escape sequence to abort.

    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping vrf FOO 172.16.0.1 source lo1         
    Mon Nov 11 22:40:35.763 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms
    RP/0/3/CPU0:XR2#trace  10.0.0.1 source 10.0.0.20
    Mon Nov 11 22:40:57.792 UTC

    Type escape sequence to abort.
    Tracing the route to 10.0.0.1

     1  10.19.20.19 [MPLS: Label 16033 Exp 0] 6 msec  6 msec  5 msec
     2  10.0.193.3 [MPLS: Label 17 Exp 0] 2 msec  3 msec  2 msec
     3  10.0.13.1 2 msec  *  2 msec
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping vrf FOO 172.16.0.1 source lo1         
    Mon Nov 11 22:41:05.807 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/3 ms

    RP/0/3/CPU0:XR2#sh ip cef 10.0.0.1/32 de
    Mon Nov 11 22:46:50.583 UTC
    10.0.0.1/32, version 0, internal 0x40040001 (ptr 0x9cd5d838) [1], 0x0 (0x9cd2abfc), 0x4500 (0x9d5224b0)
     Updated Nov 11 22:35:26.210
     local adjacency point2point
     Prefix Len 32, traffic index 0, precedence routine (0)
      gateway array (0x9cbfec20) reference count 9, flags 0xd00, source lsd (2),
                    [4 type 5 flags 0x101001 (0x9d554f14) ext 0x0 (0x0)]
      LW-LDI[type=5, refc=3, ptr=0x9cd2abfc, sh-ldi=0x9d554f14]
       via 10.19.20.19, POS0/7/0/0, 7 dependencies, weight 0, class 0 [flags 0x0]
        next hop 10.19.20.19
        local adjacency
         local label 16026      labels imposed {16033}


        Load distribution: 0 (refcount 4)

        Hash  OK  Interface                 Address
        0     Y   POS0/7/0/0                point2point   
    RP/0/3/CPU0:XR2#

     

    XR1

    interface POS0/6/0/0
     ipv4 address 10.19.20.19 255.255.255.0
     ipv6 address 2001:10:19:20::19/64
    !

    RP/0/0/CPU0:XR1#sh mpls for
    Mon Nov 11 22:47:24.117 UTC
    Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
    Label  Label       or ID              Interface                    Switched   
    ------ ----------- ------------------ ------------ --------------- ------------
    16001  Pop         PW(10.0.0.2:100)   Gi0/1/0/1.100 point2point     153272     
    16002  Pop         PW(10.0.0.2:200)   Gi0/1/0/1.200 point2point     153340     
    16005  Pop         10.0.193.3/32      Gi0/1/0/0.193 10.0.193.3      20963      
    16018  Pop         10.0.0.20/32       PO0/6/0/0    10.19.20.20     13601      
    16020  Pop         10.0.205.0/24      Gi0/1/0/0.195 10.0.195.5      0          
           Pop         10.0.205.0/24      PO0/6/0/0    10.19.20.20     0          
    16025  Pop         10.0.220.0/24      PO0/6/0/0    10.19.20.20     0          
    16026  Pop         10.0.0.5/32        Gi0/1/0/0.195 10.0.195.5      72441      
    16029  Pop         10.0.25.0/24       Gi0/1/0/0.195 10.0.195.5      0          
    16030  16          10.0.0.2/32        Gi0/1/0/0.195 10.0.195.5      66516      
    16031  Aggregate   FOO: Per-VRF Aggr[V]  
                                          FOO                          0          
    16032  Aggregate   FOO: Per-VRF Aggr[V]  
                                          FOO                          0          
    16033  17          10.0.0.1/32        Gi0/1/0/0.193 10.0.193.3      142950     
    16034  Pop         10.0.0.3/32        Gi0/1/0/0.193 10.0.193.3      1040       
    16035  16          10.0.0.4/32        Gi0/1/0/0.193 10.0.193.3      17494      
    RP/0/0/CPU0:XR1#
    RP/0/0/CPU0:XR1#sh bgp ipv4 labeled-unicast labels
    Mon Nov 11 22:47:46.166 UTC
    BGP router identifier 10.0.0.19, local AS number 1000
    BGP generic scan interval 60 secs
    BGP table state: Active
    Table ID: 0xe0000000
    BGP main routing table version 26
    BGP scan interval 60 secs

    Status codes: s suppressed, d damped, h history, * valid, > best
                  i - internal, r RIB-failure, S stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
       Network            Next Hop        Rcvd Label      Local Label
    *> 10.0.0.1/32        10.0.193.3      17              16033          
    *> 10.0.0.2/32        10.0.195.5      nolabel         16030          
    *> 10.0.0.3/32        10.0.193.3      3               16034          
    *> 10.0.0.4/32        10.0.193.3      16              16035          
    *> 10.0.0.5/32        10.0.195.5      nolabel         16026          
    *> 10.0.0.19/32       0.0.0.0         nolabel         3              
    *> 10.0.0.20/32       10.19.20.20     nolabel         16018          

    Processed 7 prefixes, 7 paths
    RP/0/0/CPU0:XR1#

     


  • I isolated the issue to the POS subinterfaces between XR1
    and XR2. I you change the configuration to avoid frame-relay subinterface, the issue
    does not happen:

    XR 2

    RP/0/3/CPU0:XR2#sh run inter poS 0/7/0/0
    Mon Nov 11 22:39:59.561 UTC
    interface POS0/7/0/0
     ipv4 address 10.19.20.20 255.255.255.0
     ipv6 address 2001:10:19:20::20/64
    !

    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping mpls ipv4  10.0.0.1/32 source 10.0.0.20
    Mon Nov 11 22:40:11.166 UTC

    Sending 5, 100-byte MPLS Echos to 10.0.0.1/32,
          timeout is 2 seconds, send interval is 0 msec:

    Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
      'L' - labeled output interface, 'B' - unlabeled output interface,
      'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
      'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
      'P' - no rx intf label prot, 'p' - premature termination of LSP,
      'R' - transit router, 'I' - unknown upstream index,
      'X' - unknown return code, 'x' - return code 0

    Type escape sequence to abort.

    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping vrf FOO 172.16.0.1 source lo1         
    Mon Nov 11 22:40:35.763 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms
    RP/0/3/CPU0:XR2#trace  10.0.0.1 source 10.0.0.20
    Mon Nov 11 22:40:57.792 UTC

    Type escape sequence to abort.
    Tracing the route to 10.0.0.1

     1  10.19.20.19 [MPLS: Label 16033 Exp 0] 6 msec  6 msec  5 msec
     2  10.0.193.3 [MPLS: Label 17 Exp 0] 2 msec  3 msec  2 msec
     3  10.0.13.1 2 msec  *  2 msec
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#ping vrf FOO 172.16.0.1 source lo1         
    Mon Nov 11 22:41:05.807 UTC
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 172.16.0.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/3 ms

    RP/0/3/CPU0:XR2#
    RP/0/3/CPU0:XR2#sh ip cef 10.0.0.1/32 de
    Mon Nov 11 22:46:50.583 UTC
    10.0.0.1/32, version 0, internal 0x40040001 (ptr 0x9cd5d838) [1], 0x0 (0x9cd2abfc), 0x4500 (0x9d5224b0)
     Updated Nov 11 22:35:26.210
     local adjacency point2point
     Prefix Len 32, traffic index 0, precedence routine (0)
      gateway array (0x9cbfec20) reference count 9, flags 0xd00, source lsd (2),
                    [4 type 5 flags 0x101001 (0x9d554f14) ext 0x0 (0x0)]
      LW-LDI[type=5, refc=3, ptr=0x9cd2abfc, sh-ldi=0x9d554f14]
       via 10.19.20.19, POS0/7/0/0, 7 dependencies, weight 0, class 0 [flags 0x0]
        next hop 10.19.20.19
        local adjacency
         local label 16026      labels imposed {16033}


        Load distribution: 0 (refcount 4)

        Hash  OK  Interface                 Address
        0     Y   POS0/7/0/0                point2point   
    RP/0/3/CPU0:XR2#

     

    XR 1

    RP/0/0/CPU0:XR1#sh run inter poS 0/6/0/0
    Mon Nov 11 22:48:33.390 UTC
    interface POS0/6/0/0
     ipv4 address 10.19.20.19 255.255.255.0
     ipv6 address 2001:10:19:20::19/64
    !

    RP/0/0/CPU0:XR1#sh mpls for
    Mon Nov 11 22:47:24.117 UTC
    Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes      
    Label  Label       or ID              Interface                    Switched   
    ------ ----------- ------------------ ------------ --------------- ------------
    16001  Pop         PW(10.0.0.2:100)   Gi0/1/0/1.100 point2point     153272     
    16002  Pop         PW(10.0.0.2:200)   Gi0/1/0/1.200 point2point     153340     
    16005  Pop         10.0.193.3/32      Gi0/1/0/0.193 10.0.193.3      20963      
    16018  Pop         10.0.0.20/32       PO0/6/0/0    10.19.20.20     13601      
    16020  Pop         10.0.205.0/24      Gi0/1/0/0.195 10.0.195.5      0          
           Pop         10.0.205.0/24      PO0/6/0/0    10.19.20.20     0          
    16025  Pop         10.0.220.0/24      PO0/6/0/0    10.19.20.20     0          
    16026  Pop         10.0.0.5/32        Gi0/1/0/0.195 10.0.195.5      72441      
    16029  Pop         10.0.25.0/24       Gi0/1/0/0.195 10.0.195.5      0          
    16030  16          10.0.0.2/32        Gi0/1/0/0.195 10.0.195.5      66516      
    16031  Aggregate   FOO: Per-VRF Aggr[V]  
                                          FOO                          0          
    16032  Aggregate   FOO: Per-VRF Aggr[V]  
                                          FOO                          0          
    16033  17          10.0.0.1/32        Gi0/1/0/0.193 10.0.193.3      142950     
    16034  Pop         10.0.0.3/32        Gi0/1/0/0.193 10.0.193.3      1040       
    16035  16          10.0.0.4/32        Gi0/1/0/0.193 10.0.193.3      17494      
    RP/0/0/CPU0:XR1#
    RP/0/0/CPU0:XR1#
    RP/0/0/CPU0:XR1#
    RP/0/0/CPU0:XR1#sh bgp ipv4 labeled-unicast labels
    Mon Nov 11 22:47:46.166 UTC
    BGP router identifier 10.0.0.19, local AS number 1000
    BGP generic scan interval 60 secs
    BGP table state: Active
    Table ID: 0xe0000000
    BGP main routing table version 26
    BGP scan interval 60 secs

    Status codes: s suppressed, d damped, h history, * valid, > best
                  i - internal, r RIB-failure, S stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
       Network            Next Hop        Rcvd Label      Local Label
    *> 10.0.0.1/32        10.0.193.3      17              16033          
    *> 10.0.0.2/32        10.0.195.5      nolabel         16030          
    *> 10.0.0.3/32        10.0.193.3      3               16034          
    *> 10.0.0.4/32        10.0.193.3      16              16035          
    *> 10.0.0.5/32        10.0.195.5      nolabel         16026          
    *> 10.0.0.19/32       0.0.0.0         nolabel         3              
    *> 10.0.0.20/32       10.19.20.20     nolabel         16018          

    Processed 7 prefixes, 7 paths
    RP/0/0/CPU0:XR1#

     

     

Sign In or Register to comment.