Lets talk eBGP multihop/TTL security/connectivity check

So I did some digging, around, as the concept of eBGP multhop vs TTL security just did not make sense to me.

Using this topology...(each device has a lo0 ip address--1.1.1.1 through 3.3.3.3)

image

 

Now, my understanding was that if we wish to create a eBGP neighbor relationship using loopbacks, we have to use eBGP multihop.  My understanding was that this is necessary because we will decrement the TTL by 1 and it will get dropped.

Hm..

Then how come I can create a neighbor relationship from Mario (1.1.1.) to Bowser (3.3.3.3) with eBGP multihop of 2 configured?

So with the default eBGP mutihop of 1 configured, if you do a show ip bgp neighbor 2.2.2.2 on Mario, you will see the following:

"External BGP neighbor not directly connected."  

What if I change the ebgp multihop value to 2?

"Connection is ECN Disabled"

Ok.......with that logic, that means that any ebgp multihop value >1 wiill auto-disable the connectiivty check.  But that isn't fair..why can't we configure an eBGP multhop value of 1 to disable the connectivity check?  Well, if the device is only one hop away, you can manually disable the connectivity check "neighbor 2.2.2.2 disable-connected-check"  on Mario and "neighbor 2.2.2.2 disable-connected-check" on Luigi essentially does the same as eBGP multihop with a value > or equal to 2 (while keeping the default outgoing TTL).  Alternatively, you can set the ttl-security path value to 2 (using 1 isn't even allowed...).

 

Now...TTL security.

Something interesting about TTL security.  By default, the outgoing TTL for an eBGP neighbor relationship is only 1.  We can override this with eBGP multhop or TTL security.  How's that?

From Mario's perspective to get to Bowser:

"Connection is ECN Disabled, Mininum incoming TTL 253, Outgoing TTL 255"

So Mario doesn't care about its own outgoing MTU (which is set for 255)..but more so cares about the incoming MTU.  By configuring both for a TTL-security path value of 2, it further proves that using the loopback as the TCP connection source isn't what is breaking the neighbor relationship....but the connectivity check instead (also disabled when TTL-Security is enabled).

 

I feel as if I may have rambled, but hey..if you were in the same boat as me, it might help.

 

Kyle

 

Comments

  • I have made the same findings in the past.

    So with disable-connected-check, either it ignores the expired TTL or the packets is sent to the BGP process before the TTL expires. I'm not sure how it works internally in the code, either way disable-connected-check is enough to get peering up between loopbacks on eBGP.

    This is a pretty common misconception that you MUST set ebgp-multihop to 2, which is not the case.

    I wrote about it at my blog a couple of years back:

    http://lostintransit.se/2012/09/14/some-important-details-of-bgp/

  • Hi Kyle,

    First thing to say is that eBGP multihop and TTL security can't be used at the same time, they are mutually exclusive. There are some differences between the two feature that deserve mentioning. Consider the following scenario : 

    image

     

    R5 needs to establish an eBGP neighbor relationship with R4's loopback 150.1.4.4.

    Let's see how we can do that.

     

    CASE I : eBGP multihop

    router bgp 5

     bgp log-neighbor-changes

     neighbor 150.1.4.4 remote-as 14

     neighbor 150.1.4.4 ebgp-multihop 255

     neighbor 150.1.4.4 update-source Loopback0

     

    R5#sh ip bgp neigh 150.1.4.4 | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255

     

    At this moment, R5 has eBGP multihop configured at default value, meaning that the TCP packet used in BGP negotiation will have an outgoing  TTL of 255 hops, so 150.1.4.4 can be 255 hops away. Since in this case it is only 2 hops away, the peer neighbor relationship comes UP.

    R5#traceroute 150.1.4.4

    Type escape sequence to abort.

    Tracing the route to 150.1.4.4

    VRF info: (vrf in name/id, vrf out name/id)

      1 155.1.45.4 1 msec *  2 msec

     

    CASE II : disable-connected-check

    router bgp 5

     bgp log-neighbor-changes

     neighbor 150.1.4.4 remote-as 14

     neighbor 150.1.4.4 disable-connected-check

     neighbor 150.1.4.4 update-source Loopback0

     


    R5#sh ip bgp neigh 150.1.4.4 | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1


    On R4, I have left ebgp-multihop configured, configuration works as expected, as TTL "rule" is accomplished :




    R4#sh run | s r bgp

    router bgp 14

     bgp log-neighbor-changes

     neighbor 150.1.5.5 remote-as 5

     neighbor 150.1.5.5 ebgp-multihop 255

     neighbor 150.1.5.5 update-source Loopback0

    R4#

    R4#sh ip bgp nei 150.1.5.5 | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255



    CASE III : eBGP multihop with specific hop definition

     

    R5 :

    router bgp 5

     bgp log-neighbor-changes

     neighbor 150.1.4.4 remote-as 14

     neighbor 150.1.4.4 ebgp-multihop 2

     neighbor 150.1.4.4 update-source Loopback0

     

    R5#sh ip bgp neighbors 150.1.4.4 | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 2

     

    R4 :

    router bgp 14

     bgp log-neighbor-changes

     neighbor 150.1.5.5 remote-as 5

     neighbor 150.1.5.5 ebgp-multihop 2

     neighbor 150.1.5.5 update-source Loopback0

     

    R4#sh ip bgp neighbors 150.1.5.5 | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 2


    Now suppose that the direct link between R4 and R5 is down. The eBGP peering is rerouted through R3 and the connection remains UP :






    R5#traceroute 150.1.4.4 sou loo0

    Type escape sequence to abort.

    Tracing the route to 150.1.4.4

    VRF info: (vrf in name/id, vrf out name/id)

      1 155.1.0.3 5 msec 5 msec 6 msec

      2 155.1.37.7 6 msec 8 msec 8 msec

      3 155.1.67.6 5 msec 10 msec 6 msec

      4 155.1.146.4 6 msec *  6 msec





    R5#sh ip bgp neigh | i BGP

    BGP neighbor is 150.1.4.4,  remote AS 14, external link

      BGP version 4, remote router ID 150.1.4.4

      BGP state = Established, up for 00:01:56

      BGP table version 1, neighbor version 1/0

      External BGP neighbor may be up to 2 hops away.







    CASE IV : eBGP ttl-security




    R5 : 




    router bgp 5

     bgp log-neighbor-changes

     timers bgp 5 15 15

     neighbor 150.1.4.4 remote-as 14

     neighbor 150.1.4.4 ttl-security hops 2

     neighbor 150.1.4.4 update-source Loopback0




    R5#sh ip bgp neighbors | i TTL

    Connection is ECN Disabled, Mininum incoming TTL 253, Outgoing TTL 255




    R5 is now saying that the neighbor MUST be at most 2 hops away : 


    Mininum incoming TTL 253




    Now, R5 still has a route to R4's loopback, but it's more than two hops away : 




    R5#traceroute 150.1.4.4 sou loo0

    Type escape sequence to abort.

    Tracing the route to 150.1.4.4

    VRF info: (vrf in name/id, vrf out name/id)

      1 155.1.0.3 5 msec 6 msec 5 msec

      2 155.1.37.7 6 msec 5 msec 5 msec

      3 155.1.67.6 6 msec 6 msec 5 msec

      4 155.1.146.4 7 msec *  6 msec





    As the minimum TTL to accept is 253, the connection will not come UP in this case. Starting from R4, TTL is 255 and when it reaches R5 will be 251, so it will not be an accepted value. The BGP connection will be torn DOWN : 




    R5#debug ip tcp packet 

    TCP Packet debugging is on


    TCP0: bad seg from 150.1.4.4 -- Discard the invalid TTL packets, packet ttl 251: port 179 seq 3417262799 ack 0 rcvnxt 0 rcvwnd 16384 len 0




    R5#

    %BGP-3-NOTIFICATION: sent to neighbor 150.1.4.4 4/0 (hold time expired) 0 bytes 


    %BGP-5-NBR_RESET: Neighbor 150.1.4.4 reset (BGP Notification sent)

    %BGP-5-ADJCHANGE: neighbor 150.1.4.4 Down BGP Notification sent

    %BGP_SESSION-5-ADJCHANGE: neighbor 150.1.4.4 IPv4 Unicast topology base removed from session  BGP Notification sent

    R5#





    Suppose now that we want to increase the TTL on R5 so that the TCP connection will be accepted on R4 :



    R5(config-router)#neigh 150.1.4.4 ebgp-multihop 

    Remove ttl-security before configuring ebgp-multihop





    As I mentioned at the very beginning of this post, these two features are mutually exclusive and can NOT be configured on a device at the same time.




    The difference between the two features is that with ebgp-multihop you set the outgoing TTL to the specific value, but with ttl-security you set the minimum incoming TTL to be accepted. TTL-security prevents a rogue device to be inserted in the middle of the network for hijacking purposes and also prevents rerouting the eBGP session through another path if available (longer path).






  • Good responces all around!  Thanks for the posts everyone!  I'll checked out your blog in the past daniel--definitely good information.

  • Thinking about it some more....could one argue that the optimal design for directly connected devices using loopbacks would be:

     

    1. Disable connectivity check

    2. Configure TTL-security for 1 hop

     

    That way we don't care about the connected device...but it has to be local to the peering router to allow BGP connectivity?

  • Hi Kyle, 

    It depends on what you're referring to when saying optimal.

    Real world scenarios typically use directly connected interface's IP address for eBGP and loopback IP address for iBGP, making use of peer authentication also.



    On 27 Jul 2014, at 20:07, kylecbarnes <[email protected]> wrote:

    Thinking about it some more....could one argue that the optimal design for directly connected devices using loopbacks would be:

     

    1. Disable connectivity check

    2. Configure TTL-security for 1 hop

     

    That way we don't care about the connected device...but it has to be local to the peering router to allow BGP connectivity?




    INE - The Industry Leader in CCIE Preparation

    http://www.INE.com



    Subscription information may be found at:

    http://www.ieoc.com/forums/ForumSubscriptions.aspx
  • I can't think of a reason you'd want to EBGP peer to the loopback of a directly connected device.  Generally, if multihop has to be done at all, it's to overcome some other shortcoming in design, so calling any scenario like that "optimal" is kind of a misnomer (IMO/IME/YMMV/etc).

    You'd also probably want to consider what you were trying to protect against.  Kind of like paswording EBGP, some things that would seem to make sense can actually be used against you.

Sign In or Register to comment.