Hands-on BGP lab setup

Adam Tyler

Hey Ronnie, I am going through your hands-on BGP class and had a quick question on setup. So far I only have R1,R2,R3, and R4 going. I've deployed OSPF for R1,R2, and R3, then implemented iBGP per your direction in the class. That all seems to be working fine.

I setup a peering between R2 and R4 now over 172.16.24.0/24. So this is my first eBGP peering and I am a little confused. So because OSPF is an IGP protocol, I assume I am not supposed to include R4 into the OSFP process. Am I supposed to static route R1 and R3 to the R4 peer IP to bring up those BGP peers?

R1 route, bgp summary, and show BGP

Ronnie Wong

@Adam-Tyler,

So the issue as you see in the peering, is that R1 thinks that R4 is directly connected. But the diagram clearly shows it's not. So the peering between R2 and R1 is probably the issue. There is a command, I'm trying to hint at...without telling you! ;). only because I want to see if you notice it.

Let me know if you don't remember it.

Adam Tyler

@Ronnie-Wong Ya, not sure what you were hinting at there, sorry. Still trying to wrap my head around BGP, it is a lot different than OSPF which I've worked with in the field. In the screenshots above R1 and R3 are not peered with R4 at all, simply because neither of them had a route to the 172.16.24.0/24 network, I didn't even issue peering commands for those routers until I figured out how to deal with that. Was planning to peer each with 172.16.24.4.

I did get everything talking, however I don't think it is how you accomplished the 4 router setup in your demonstration. Basically I used R2 to passively inject the 172.16.24.0/24 route into the OSPF process. Once R1 and R3 knew how to get to 172.16.24.4, I was able to create peering relationships between R4 and R1/R3. Had to leverage the ebgp-multihop command. Current config status of my lab at the end of this post for reference.

One snag I ran into after all peering was online was on R1 attempting to issue the ssh 4.4.4.4 command just to get the CLI of R4. Turned out that R1 was using the default Gi0/0 interface IP to get there and R4 didn't have a route for 10.0.12.0/24. I used BGP on R2 to advertise it and I could then SSH to R4.

BGP is so strange, if you look at the R3 show bgp output after I used R2 to inject 10.0.12.0/24 into the BGP process, it has an r>i entry and a next hop of 2.2.2.2. A router that couldn't possible serve as a next hop for R3 to that network. It didn't make it into the routing table because of rib failure and it serves R4 okay. Just strange. Don't really understand it yet. OSPF always would propagate the next hop IP correctly. It seems that BGP has the ability to hand out bad next hops all over the place.

R4 has three entries for 10.0.12.0/24 in it's show bgp output. It selected valid best for 172.16.24.2, I guess because that is the only one that would work? How did it know not to select one of the routes with a bad next hop?

R4 show bgp snippet:
 *    10.0.12.0/24     3.3.3.3                                0 65123 i <R4 not directly connected to this router.
 *                     1.1.1.1                                0 65123 i <R4 not directly connected to this router.
 *>                    172.16.24.2              0             0 65123 i <How did R4 know to select this one?

R1#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.12.1       YES NVRAM  up                    up      
GigabitEthernet0/1         10.0.13.1       YES NVRAM  up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Serial0/1/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  1.1.1.1         YES NVRAM  up                    up      
R1#!

R1#show run | s ospf
router ospf 1
 router-id 1.1.1.1
 auto-cost reference-bandwidth 40000
 network 1.1.1.1 0.0.0.0 area 0
 network 10.0.12.0 0.0.0.255 area 0
 network 10.0.13.0 0.0.0.255 area 0

R1#!
R1#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 neighbor 2.2.2.2 remote-as 65123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 ebgp-multihop 2
 neighbor 172.16.24.4 update-source Loopback0

R1#!
R1#show bgp
BGP table version is 7, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   1.1.1.1/32       0.0.0.0                  0         32768 i
 r>i  2.2.2.2/32       2.2.2.2                  0    100      0 i
 r>i  3.3.3.3/32       3.3.3.3                  0    100      0 i
 * i  4.4.4.4/32       172.16.24.4              0    100      0 65004 i
 *>                    172.16.24.4              0             0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 r>i  10.0.12.0/24     2.2.2.2                  0    100      0 i

R1#!
R1#show bgp sum
BGP router identifier 1.1.1.1, local AS number 65123
BGP table version is 7, main routing table version 7
5 network entries using 720 bytes of memory
7 path entries using 616 bytes of memory
4/3 BGP path/bestpath attribute entries using 640 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2000 total bytes of memory
BGP activity 11/6 prefixes, 44/37 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4        65123      24      23        7    0    0 00:17:21        3
3.3.3.3         4        65123      23      23        7    0    0 00:17:15        2
172.16.24.4     4        65004      28      26        7    0    0 00:16:11        1

R1#!
R1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
C        1.1.1.1 is directly connected, Loopback0
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/41] via 10.0.12.2, 01:17:29, GigabitEthernet0/0
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/41] via 10.0.13.3, 01:17:29, GigabitEthernet0/1
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:16:13
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C        10.0.12.0/24 is directly connected, GigabitEthernet0/0
L        10.0.12.1/32 is directly connected, GigabitEthernet0/0
C        10.0.13.0/24 is directly connected, GigabitEthernet0/1
L        10.0.13.1/32 is directly connected, GigabitEthernet0/1
      172.16.0.0/24 is subnetted, 1 subnets
O        172.16.24.0 [110/80] via 10.0.12.2, 00:46:24, GigabitEthernet0/0


R2#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.12.2       YES NVRAM  up                    up      
GigabitEthernet0/1         172.16.24.2     YES NVRAM  up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  2.2.2.2         YES NVRAM  up                    up      

R2#!
R2#show run | s ospf
router ospf 1
 router-id 2.2.2.2
 auto-cost reference-bandwidth 40000
 passive-interface GigabitEthernet0/1
 network 2.2.2.0 0.0.0.255 area 0
 network 10.0.12.0 0.0.0.255 area 0
 network 172.16.24.0 0.0.0.255 area 0

R2#!
R2#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 2.2.2.2 mask 255.255.255.255
 network 10.0.12.0 mask 255.255.255.0
 network 10.0.13.0 mask 255.255.255.0
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 update-source GigabitEthernet0/1

R2#!
R2#show bgp
BGP table version is 7, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>i  1.1.1.1/32       1.1.1.1                  0    100      0 i
 *>   2.2.2.2/32       0.0.0.0                  0         32768 i
 r>i  3.3.3.3/32       3.3.3.3                  0    100      0 i
 * i  4.4.4.4/32       172.16.24.4              0    100      0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 *>                    172.16.24.4              0             0 65004 i
 *>   10.0.12.0/24     0.0.0.0                  0         32768 i
 *>   10.0.13.0/24     10.0.12.1               80         32768 i

R2#!
R2#show bgp sum
BGP router identifier 2.2.2.2, local AS number 65123
BGP table version is 7, main routing table version 7
6 network entries using 864 bytes of memory
8 path entries using 704 bytes of memory
5/4 BGP path/bestpath attribute entries using 800 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2392 total bytes of memory
BGP activity 12/6 prefixes, 44/36 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123      23      25        7    0    0 00:17:36        2
3.3.3.3         4        65123      23      24        7    0    0 00:17:24        2
172.16.24.4     4        65004      28      25        7    0    0 00:17:36        1

R2#!
R2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/41] via 10.0.12.1, 01:17:38, GigabitEthernet0/0
      2.0.0.0/32 is subnetted, 1 subnets
C        2.2.2.2 is directly connected, Loopback0
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/81] via 10.0.12.1, 01:17:38, GigabitEthernet0/0
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:16:32
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.0.12.0/24 is directly connected, GigabitEthernet0/0
L        10.0.12.2/32 is directly connected, GigabitEthernet0/0
O        10.0.13.0/24 [110/80] via 10.0.12.1, 01:17:38, GigabitEthernet0/0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.24.0/24 is directly connected, GigabitEthernet0/1
L        172.16.24.2/32 is directly connected, GigabitEthernet0/1


R3#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.13.3       YES NVRAM  up                    up      
GigabitEthernet0/1         192.168.35.3    YES NVRAM  down                  down    
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Serial0/1/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  3.3.3.3         YES NVRAM  up                    up      
R3#!
R3#show run | s ospf
router ospf 1
 router-id 3.3.3.3
 auto-cost reference-bandwidth 40000
 network 3.3.3.3 0.0.0.0 area 0
 network 10.0.13.0 0.0.0.255 area 0

R3#!
R3#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 3.3.3.3 mask 255.255.255.255
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 2.2.2.2 remote-as 65123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 ebgp-multihop 3
 neighbor 172.16.24.4 update-source Loopback0

R3#!
R3#show bgp
BGP table version is 8, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>i  1.1.1.1/32       1.1.1.1                  0    100      0 i
 r>i  2.2.2.2/32       2.2.2.2                  0    100      0 i
 *>   3.3.3.3/32       0.0.0.0                  0         32768 i
 *>   4.4.4.4/32       172.16.24.4              0             0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 r>i  10.0.12.0/24     2.2.2.2                  0    100      0 i
 r>i  10.0.13.0/24     10.0.12.1               80    100      0 i

R3#!
R3#show bgp sum
BGP router identifier 3.3.3.3, local AS number 65123
BGP table version is 8, main routing table version 8
6 network entries using 864 bytes of memory
8 path entries using 704 bytes of memory
5/4 BGP path/bestpath attribute entries using 800 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2392 total bytes of memory
BGP activity 11/5 prefixes, 33/25 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123      24      24        8    0    0 00:17:53        2
2.2.2.2         4        65123      24      24        8    0    0 00:17:47        4
172.16.24.4     4        65004      28      26        8    0    0 00:16:24        1

R3#!
R3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/41] via 10.0.13.1, 01:17:57, GigabitEthernet0/0
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/81] via 10.0.13.1, 01:17:57, GigabitEthernet0/0
      3.0.0.0/32 is subnetted, 1 subnets
C        3.3.3.3 is directly connected, Loopback0
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:16:24
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O        10.0.12.0/24 [110/80] via 10.0.13.1, 01:17:57, GigabitEthernet0/0
C        10.0.13.0/24 is directly connected, GigabitEthernet0/0
L        10.0.13.3/32 is directly connected, GigabitEthernet0/0
      172.16.0.0/24 is subnetted, 1 subnets
O        172.16.24.0 [110/120] via 10.0.13.1, 00:45:44, GigabitEthernet0/0


R4#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         172.16.24.4     YES NVRAM  up                    up      
GigabitEthernet0/1         unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Loopback0                  4.4.4.4         YES NVRAM  up                    up      

R4#!
R4#show run | s ospf

R4#!
R4#show run | s bgp
router bgp 65004
 bgp log-neighbor-changes
 network 4.4.4.4 mask 255.255.255.255
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 ebgp-multihop 2
 neighbor 1.1.1.1 update-source GigabitEthernet0/0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 ebgp-multihop 3
 neighbor 3.3.3.3 update-source GigabitEthernet0/0
 neighbor 172.16.24.2 remote-as 65123
 neighbor 172.16.24.2 update-source GigabitEthernet0/0

R4#!
R4#show bgp
BGP table version is 17, local router ID is 4.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *    1.1.1.1/32       3.3.3.3                                0 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *>                    172.16.24.2                            0 65123 i
 *    2.2.2.2/32       3.3.3.3                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    172.16.24.2              0             0 65123 i
 *    3.3.3.3/32       1.1.1.1                                0 65123 i
 *                     3.3.3.3                  0             0 65123 i
 *>                    172.16.24.2                            0 65123 i
 *>   4.4.4.4/32       0.0.0.0                  0         32768 i
 *    10.0.12.0/24     3.3.3.3                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    172.16.24.2              0             0 65123 i
 *>   10.0.13.0/24     3.3.3.3                                0 65123 i
 *                     172.16.24.2             80             0 65123 i

------------------------------

R4#!
R4#show bgp sum
BGP router identifier 4.4.4.4, local AS number 65004
BGP table version is 17, main routing table version 17
6 network entries using 864 bytes of memory
15 path entries using 1320 bytes of memory
4/3 BGP path/bestpath attribute entries using 640 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2848 total bytes of memory
BGP activity 13/7 prefixes, 79/64 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123      27      29       17    0    0 00:17:13        4
3.3.3.3         4        65123      27      28       17    0    0 00:16:46        5
172.16.24.2     4        65123      26      29       17    0    0 00:18:22        5

R4#!
R4#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
B        1.1.1.1 [20/0] via 172.16.24.2, 00:17:18
      2.0.0.0/32 is subnetted, 1 subnets
B        2.2.2.2 [20/0] via 172.16.24.2, 00:17:18
      3.0.0.0/32 is subnetted, 1 subnets
B        3.3.3.3 [20/0] via 172.16.24.2, 00:16:47
      4.0.0.0/32 is subnetted, 1 subnets
C        4.4.4.4 is directly connected, Loopback0
      10.0.0.0/24 is subnetted, 2 subnets
B        10.0.12.0 [20/0] via 172.16.24.2, 00:06:05
B        10.0.13.0 [20/0] via 3.3.3.3, 00:02:51
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.24.0/24 is directly connected, GigabitEthernet0/0
L        172.16.24.4/32 is directly connected, GigabitEthernet0/0

Ronnie Wong

@Adam-Tyler,

Good working through...

the command option for you to take a look at next-hop self You need R2 to tell R1 for BGP routes you learn, to change the next hop to R2 so it can route back to R4.

Adam Tyler

@Ronnie-Wong Thanks Ronnie. In the case of using the next-hop-self command on R2.. Let's say you didn't want R2 to be the next hop for all routes, only 1. Is there a use case for that? Some route-map trick for that? Otherwise next-hop-self appears to be global for all BGP routes on R2 ?

I spent a bunch of time in the lab last night and added router 5 and 6. Everything seems to route to everything and I didn't end up having to use next-hop-self. However, not sure I would have achieved this result if I didn't fully control every router in every AS.

Take for example my below configs with R5 and R6. R6 is learning all routes through eBGP and there is no IGP protocol running. show bgp on R6 shows the valid/best route selected for 1.1.1.1 as...

R6
*> 1.1.1.1/32       172.16.26.2                            0 65123 i
*                   192.168.56.5                           0 65005 65123 i

Now I assume it is using 172.16.26.2 because of the AS Path which makes sense. 65123 instead of 65005 65123 as with 192.168.56.5.

Looks like I will be able to play with AS Path manipulation in adjusting the route for R3, for example...

R6
*> 3.3.3.3/32       172.16.26.2                            0 65123 i
*                   192.168.56.5                           0 65005 65123 i

Although the gateway 172.16.26.2 is a shorter AS Path, there are more hops/routers going that way.

So couple of follow up questions after labing all of this if you don't mind...

is an appropriate use of iBGP on the public internet in your own AS? For example, would this config be valid if all the AS numbers were ARIN issued and public IP addressing I actually owned?
If iBGP is something you can use on the public internet, is it typical to peer every single iBGP router in our AS to the directly connected public eBGP AS routers? In this lab for example the peering between R5 and R1,R2,R3. Or do you typically only peer your internal (iBGP) routers directly connected to the ISPs? IE.. only peer R5 with R2 and R3. Or only peer R6 with R2.
Next hop IP in show bgp is sometimes very confusing. take for example in my lab R5. It has an entry in show bgp for 4.4.4.4 as follows..

R5
 *    4.4.4.4/32       192.168.56.6                           0 65006 65123 65004 i
     
      *                     192.168.35.3                           0 65123 65004 i
     
      *                     1.1.1.1                                0 65123 65004 i
     
      *>                    2.2.2.2                                0 65123 65004 i`

As you can see, it selected 2.2.2.2 as the Next Hop. Even if you look at the routing table of R5, the gateway IP is 2.2.2.2. This breaks my brain. I thought the Next Hop had to be an IP that was directly connected to the router in some way.

R5#show ip route
      4.0.0.0/32 is subnetted, 1 subnets
      
      B        4.4.4.4 [20/0] via 2.2.2.2, 00:48:18

But I can ping from R5 to 4.4.4.4 all day long... Why? Because the routing tables are complete and the R5 router knows how to get to the IP of 2.2.2.2 ?

R5
GigabitEthernet0/0         192.168.35.5    YES NVRAM  up                    up
GigabitEthernet0/1         192.168.56.5    YES NVRAM  up                    up
Loopback0                  5.5.5.5         YES NVRAM  up                    up

R5#ping 4.4.4.4 source gi0/0
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 192.168.35.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

R5#ping 4.4.4.4 source gi0/1
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 192.168.56.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

R5#ping 4.4.4.4 source lo0
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 5.5.5.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

Adam Tyler

Last post was too long, configs R1-R3 for reference...

terminal length 511
R1#!
R1#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.12.1       YES NVRAM  up                    up      
GigabitEthernet0/1         10.0.13.1       YES NVRAM  up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Serial0/1/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  1.1.1.1         YES NVRAM  up                    up      
R1#!
R1#show run | s ospf
router ospf 1
 router-id 1.1.1.1
 auto-cost reference-bandwidth 40000
 network 1.1.1.1 0.0.0.0 area 0
 network 10.0.12.0 0.0.0.255 area 0
 network 10.0.13.0 0.0.0.255 area 0
R1#!
R1#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 network 10.0.12.0 mask 255.255.255.0
 network 10.0.13.0 mask 255.255.255.0
 neighbor 2.2.2.2 remote-as 65123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 ebgp-multihop 2
 neighbor 172.16.24.4 update-source Loopback0
 neighbor 192.168.35.5 remote-as 65005
 neighbor 192.168.35.5 ebgp-multihop 2
 neighbor 192.168.35.5 update-source Loopback0
R1#!
R1#show bgp sum
BGP router identifier 1.1.1.1, local AS number 65123
BGP table version is 34, main routing table version 34
12 network entries using 1728 bytes of memory
23 path entries using 2024 bytes of memory
8/5 BGP path/bestpath attribute entries using 1280 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5128 total bytes of memory
BGP activity 13/1 prefixes, 34/11 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4        65123     113     119       34    0    0 01:31:54        8
3.3.3.3         4        65123     114     118       34    0    0 01:32:00        6
172.16.24.4     4        65004      63      65       34    0    0 00:46:17        2
192.168.35.5    4        65005      75      80       34    0    0 00:54:28        4
R1#!
R1#show bgp
BGP table version is 34, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   1.1.1.1/32       0.0.0.0                  0         32768 i
 r>i  2.2.2.2/32       2.2.2.2                  0    100      0 i
 r>i  3.3.3.3/32       3.3.3.3                  0    100      0 i
 * i  4.4.4.4/32       172.16.24.4              0    100      0 65004 i
 *>                    172.16.24.4              0             0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 *>   5.5.5.5/32       192.168.35.5             0             0 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>i  6.6.6.6/32       172.16.26.6              0    100      0 65006 i
 *                     192.168.35.5                           0 65005 65006 i
 *>   10.0.12.0/24     0.0.0.0                  0         32768 i
 * i                   2.2.2.2                  0    100      0 i
 * i  10.0.13.0/24     3.3.3.3                  0    100      0 i
 *>                    0.0.0.0                  0         32768 i
 r    172.16.24.0/24   172.16.24.4              0             0 65004 i
 r>i                   2.2.2.2                  0    100      0 i
 *>i  172.16.26.0/24   2.2.2.2                  0    100      0 i
 r    192.168.35.0     192.168.35.5             0             0 65005 i
 r>i                   3.3.3.3                  0    100      0 i
 * i  192.168.56.0     172.16.26.6              0    100      0 65006 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>                    192.168.35.5             0             0 65005 i
R1#!
R1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
C        1.1.1.1 is directly connected, Loopback0
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/41] via 10.0.12.2, 01:32:05, GigabitEthernet0/0
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/41] via 10.0.13.3, 01:32:05, GigabitEthernet0/1
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:41:28
      5.0.0.0/32 is subnetted, 1 subnets
B        5.5.5.5 [20/0] via 192.168.35.5, 00:54:32
      6.0.0.0/32 is subnetted, 1 subnets
B        6.6.6.6 [200/0] via 172.16.26.6, 00:07:04
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C        10.0.12.0/24 is directly connected, GigabitEthernet0/0
L        10.0.12.1/32 is directly connected, GigabitEthernet0/0
C        10.0.13.0/24 is directly connected, GigabitEthernet0/1
L        10.0.13.1/32 is directly connected, GigabitEthernet0/1
      172.16.0.0/24 is subnetted, 2 subnets
O        172.16.24.0 [110/80] via 10.0.12.2, 00:41:33, GigabitEthernet0/0
B        172.16.26.0 [200/0] via 2.2.2.2, 00:07:09
O     192.168.35.0/24 [110/80] via 10.0.13.3, 01:22:07, GigabitEthernet0/1
B     192.168.56.0/24 [20/0] via 192.168.35.5, 00:09:39
R1#!
R1#


R2#terminal length 511
R2#!
R2#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.12.2       YES NVRAM  up                    up      
GigabitEthernet0/1         172.16.24.2     YES NVRAM  up                    up      
GigabitEthernet0/2         172.16.26.2     YES manual up                    up      
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  2.2.2.2         YES NVRAM  up                    up      
R2#!
R2#show run | s ospf
router ospf 1
 router-id 2.2.2.2
 auto-cost reference-bandwidth 40000
 passive-interface GigabitEthernet0/1
 network 2.2.2.0 0.0.0.255 area 0
 network 10.0.12.0 0.0.0.255 area 0
 network 172.16.24.0 0.0.0.255 area 0
R2#!
R2#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 2.2.2.2 mask 255.255.255.255
 network 10.0.12.0 mask 255.255.255.0
 network 172.16.24.0 mask 255.255.255.0
 network 172.16.26.0 mask 255.255.255.0
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 update-source GigabitEthernet0/1
 neighbor 172.16.26.6 remote-as 65006
 neighbor 172.16.26.6 update-source GigabitEthernet0/2
 neighbor 192.168.35.5 remote-as 65005
 neighbor 192.168.35.5 ebgp-multihop 3
 neighbor 192.168.35.5 update-source Loopback0
R2#!
R2#show bgp sum
BGP router identifier 2.2.2.2, local AS number 65123
BGP table version is 34, main routing table version 34
12 network entries using 1728 bytes of memory
26 path entries using 2288 bytes of memory
9/5 BGP path/bestpath attribute entries using 1440 bytes of memory
5 BGP AS-PATH entries using 120 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5576 total bytes of memory
BGP activity 13/1 prefixes, 43/17 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123     120     113       34    0    0 01:32:21        6
3.3.3.3         4        65123     112     113       34    0    0 01:32:17        6
172.16.24.4     4        65004      60      65       34    0    0 00:47:55        2
172.16.26.6     4        65006      19      20       34    0    0 00:08:44        4
192.168.35.5    4        65005      77      81       34    0    0 00:57:13        4
R2#!
R2#show bgp
BGP table version is 34, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>i  1.1.1.1/32       1.1.1.1                  0    100      0 i
 *>   2.2.2.2/32       0.0.0.0                  0         32768 i
 r>i  3.3.3.3/32       3.3.3.3                  0    100      0 i
 * i  4.4.4.4/32       172.16.24.4              0    100      0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 *>                    172.16.24.4              0             0 65004 i
 *    5.5.5.5/32       172.16.26.6                            0 65006 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>                    192.168.35.5             0             0 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>   6.6.6.6/32       172.16.26.6              0             0 65006 i
 *                     192.168.35.5                           0 65005 65006 i
 * i  10.0.12.0/24     1.1.1.1                  0    100      0 i
 *>                    0.0.0.0                  0         32768 i
 r i  10.0.13.0/24     3.3.3.3                  0    100      0 i
 r>i                   1.1.1.1                  0    100      0 i
 *    172.16.24.0/24   172.16.24.4              0             0 65004 i
 *>                    0.0.0.0                  0         32768 i
 *    172.16.26.0/24   172.16.26.6              0             0 65006 i
 *>                    0.0.0.0                  0         32768 i
 r    192.168.35.0     192.168.35.5             0             0 65005 i
 r>i                   3.3.3.3                  0    100      0 i
 *>   192.168.56.0     172.16.26.6              0             0 65006 i
 * i                   192.168.35.5             0    100      0 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 *                     192.168.35.5             0             0 65005 i
R2#!
R2#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/41] via 10.0.12.1, 01:32:30, GigabitEthernet0/0
      2.0.0.0/32 is subnetted, 1 subnets
C        2.2.2.2 is directly connected, Loopback0
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/81] via 10.0.12.1, 01:32:30, GigabitEthernet0/0
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:46:50
      5.0.0.0/32 is subnetted, 1 subnets
B        5.5.5.5 [20/0] via 192.168.35.5, 00:57:17
      6.0.0.0/32 is subnetted, 1 subnets
B        6.6.6.6 [20/0] via 172.16.26.6, 00:08:49
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        10.0.12.0/24 is directly connected, GigabitEthernet0/0
L        10.0.12.2/32 is directly connected, GigabitEthernet0/0
O        10.0.13.0/24 [110/80] via 10.0.12.1, 01:32:30, GigabitEthernet0/0
      172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C        172.16.24.0/24 is directly connected, GigabitEthernet0/1
L        172.16.24.2/32 is directly connected, GigabitEthernet0/1
C        172.16.26.0/24 is directly connected, GigabitEthernet0/2
L        172.16.26.2/32 is directly connected, GigabitEthernet0/2
O     192.168.35.0/24 [110/120] via 10.0.12.1, 01:22:34, GigabitEthernet0/0
B     192.168.56.0/24 [20/0] via 172.16.26.6, 00:06:14
R2#!
R2#


R3#terminal length 511
R3#!
R3#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         10.0.13.3       YES NVRAM  up                    up      
GigabitEthernet0/1         192.168.35.3    YES NVRAM  up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Serial0/1/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  3.3.3.3         YES NVRAM  up                    up      
R3#!
R3#show run | s ospf
router ospf 1
 router-id 3.3.3.3
 auto-cost reference-bandwidth 40000
 passive-interface GigabitEthernet0/1
 network 3.3.3.3 0.0.0.0 area 0
 network 10.0.13.0 0.0.0.255 area 0
 network 192.168.35.0 0.0.0.255 area 0
R3#!
R3#show run | s bgp
router bgp 65123
 bgp log-neighbor-changes
 network 3.3.3.3 mask 255.255.255.255
 network 10.0.13.0 mask 255.255.255.0
 network 192.168.35.0
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 2.2.2.2 remote-as 65123
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 172.16.24.4 remote-as 65004
 neighbor 172.16.24.4 ebgp-multihop 3
 neighbor 172.16.24.4 update-source Loopback0
 neighbor 192.168.35.5 remote-as 65005
 neighbor 192.168.35.5 update-source GigabitEthernet0/1
R3#!
R3#show bgp sum
BGP router identifier 3.3.3.3, local AS number 65123
BGP table version is 30, main routing table version 30
12 network entries using 1728 bytes of memory
23 path entries using 2024 bytes of memory
8/5 BGP path/bestpath attribute entries using 1280 bytes of memory
4 BGP AS-PATH entries using 96 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5128 total bytes of memory
BGP activity 13/1 prefixes, 37/14 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123     119     115       30    0    0 01:32:57        6
2.2.2.2         4        65123     114     113       30    0    0 01:32:47        8
172.16.24.4     4        65004      65      66       30    0    0 00:47:14        2
192.168.35.5    4        65005      97     109       30    0    0 01:18:58        4
R3#!
R3#show bgp
BGP table version is 30, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>i  1.1.1.1/32       1.1.1.1                  0    100      0 i
 r>i  2.2.2.2/32       2.2.2.2                  0    100      0 i
 *>   3.3.3.3/32       0.0.0.0                  0         32768 i
 * i  4.4.4.4/32       172.16.24.4              0    100      0 65004 i
 *>                    172.16.24.4              0             0 65004 i
 * i                   172.16.24.4              0    100      0 65004 i
 * i  5.5.5.5/32       192.168.35.5             0    100      0 65005 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>                    192.168.35.5             0             0 65005 i
 *>i  6.6.6.6/32       172.16.26.6              0    100      0 65006 i
 *                     192.168.35.5                           0 65005 65006 i
 r>i  10.0.12.0/24     1.1.1.1                  0    100      0 i
 r i                   2.2.2.2                  0    100      0 i
 *>   10.0.13.0/24     0.0.0.0                  0         32768 i
 * i                   1.1.1.1                  0    100      0 i
 r    172.16.24.0/24   172.16.24.4              0             0 65004 i
 r>i                   2.2.2.2                  0    100      0 i
 *>i  172.16.26.0/24   2.2.2.2                  0    100      0 i
 *    192.168.35.0     192.168.35.5             0             0 65005 i
 *>                    0.0.0.0                  0         32768 i
 * i  192.168.56.0     172.16.26.6              0    100      0 65006 i
 * i                   192.168.35.5             0    100      0 65005 i
 *>                    192.168.35.5             0             0 65005 i
R3#!
R3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/41] via 10.0.13.1, 01:33:02, GigabitEthernet0/0
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/81] via 10.0.13.1, 01:32:52, GigabitEthernet0/0
      3.0.0.0/32 is subnetted, 1 subnets
C        3.3.3.3 is directly connected, Loopback0
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.24.4, 00:42:26
      5.0.0.0/32 is subnetted, 1 subnets
B        5.5.5.5 [20/0] via 192.168.35.5, 01:19:02
      6.0.0.0/32 is subnetted, 1 subnets
B        6.6.6.6 [200/0] via 172.16.26.6, 00:08:02
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O        10.0.12.0/24 [110/80] via 10.0.13.1, 01:33:02, GigabitEthernet0/0
C        10.0.13.0/24 is directly connected, GigabitEthernet0/0
L        10.0.13.3/32 is directly connected, GigabitEthernet0/0
      172.16.0.0/24 is subnetted, 2 subnets
O        172.16.24.0 [110/120] via 10.0.13.1, 00:42:31, GigabitEthernet0/0
B        172.16.26.0 [200/0] via 2.2.2.2, 00:08:07
      192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.35.0/24 is directly connected, GigabitEthernet0/1
L        192.168.35.3/32 is directly connected, GigabitEthernet0/1
B     192.168.56.0/24 [20/0] via 192.168.35.5, 00:10:36
R3#!
R3#

Adam Tyler

Last post was too long, configs R4-R6 for reference...


R4#terminal length 511
R4#!
R4#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         172.16.24.4     YES NVRAM  up                    up      
GigabitEthernet0/1         unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Loopback0                  4.4.4.4         YES NVRAM  up                    up      
R4#!
R4#show run | s ospf
R4#!
R4#show run | s bgp
router bgp 65004
 bgp log-neighbor-changes
 network 4.4.4.4 mask 255.255.255.255
 network 172.16.24.0 mask 255.255.255.0
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 ebgp-multihop 2
 neighbor 1.1.1.1 update-source GigabitEthernet0/0
 neighbor 3.3.3.3 remote-as 65123
 neighbor 3.3.3.3 ebgp-multihop 3
 neighbor 3.3.3.3 update-source GigabitEthernet0/0
 neighbor 172.16.24.2 remote-as 65123
 neighbor 172.16.24.2 update-source GigabitEthernet0/0
R4#!
R4#show bgp sum
BGP router identifier 4.4.4.4, local AS number 65004
BGP table version is 16, main routing table version 16
12 network entries using 1728 bytes of memory
35 path entries using 3080 bytes of memory
5/5 BGP path/bestpath attribute entries using 800 bytes of memory
3 BGP AS-PATH entries using 72 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5680 total bytes of memory
BGP activity 20/8 prefixes, 57/22 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123      66      65       16    0    0 00:47:41       11
3.3.3.3         4        65123      66      66       16    0    0 00:47:41       11
172.16.24.2     4        65123      66      61       16    0    0 00:48:52       11
R4#!
R4#show bgp
BGP table version is 16, local router ID is 4.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *    1.1.1.1/32       3.3.3.3                                0 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *>                    172.16.24.2                            0 65123 i
 *    2.2.2.2/32       3.3.3.3                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    172.16.24.2              0             0 65123 i
 *    3.3.3.3/32       3.3.3.3                  0             0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    172.16.24.2                            0 65123 i
 *>   4.4.4.4/32       0.0.0.0                  0         32768 i
 *    5.5.5.5/32       3.3.3.3                                0 65123 65005 i
 *                     1.1.1.1                                0 65123 65005 i
 *>                    172.16.24.2                            0 65123 65005 i
 *>   6.6.6.6/32       172.16.24.2                            0 65123 65006 i
 *                     1.1.1.1                                0 65123 65006 i
 *                     3.3.3.3                                0 65123 65006 i
 *    10.0.12.0/24     3.3.3.3                                0 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *>                    172.16.24.2              0             0 65123 i
 *    10.0.13.0/24     3.3.3.3                  0             0 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *>                    172.16.24.2                            0 65123 i
 *>   172.16.24.0/24   0.0.0.0                  0         32768 i
 *                     3.3.3.3                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *                     172.16.24.2              0             0 65123 i
 *    172.16.26.0/24   3.3.3.3                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    172.16.24.2              0             0 65123 i
 *    192.168.35.0     172.16.24.2                            0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    3.3.3.3                  0             0 65123 i
 *    192.168.56.0     172.16.24.2                            0 65123 65006 i
 *                     1.1.1.1                                0 65123 65005 i
 *>                    3.3.3.3                                0 65123 65005 i
R4#!
R4#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
B        1.1.1.1 [20/0] via 172.16.24.2, 00:47:47
      2.0.0.0/32 is subnetted, 1 subnets
B        2.2.2.2 [20/0] via 172.16.24.2, 00:47:47
      3.0.0.0/32 is subnetted, 1 subnets
B        3.3.3.3 [20/0] via 172.16.24.2, 00:47:47
      4.0.0.0/32 is subnetted, 1 subnets
C        4.4.4.4 is directly connected, Loopback0
      5.0.0.0/32 is subnetted, 1 subnets
B        5.5.5.5 [20/0] via 172.16.24.2, 00:47:47
      6.0.0.0/32 is subnetted, 1 subnets
B        6.6.6.6 [20/0] via 172.16.24.2, 00:09:46
      10.0.0.0/24 is subnetted, 2 subnets
B        10.0.12.0 [20/0] via 172.16.24.2, 00:47:47
B        10.0.13.0 [20/0] via 172.16.24.2, 00:47:47
      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.16.24.0/24 is directly connected, GigabitEthernet0/0
L        172.16.24.4/32 is directly connected, GigabitEthernet0/0
B        172.16.26.0/24 [20/0] via 172.16.24.2, 00:08:34
B     192.168.35.0/24 [20/0] via 3.3.3.3, 00:41:04
B     192.168.56.0/24 [20/0] via 3.3.3.3, 00:11:04
R4#!
R4#


R5#terminal length 511
R5#!
R5#show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
Embedded-Service-Engine0/0 unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/0         192.168.35.5    YES NVRAM  up                    up      
GigabitEthernet0/1         192.168.56.5    YES manual up                    up      
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
Serial0/0/0                unassigned      YES NVRAM  administratively down down    
Serial0/1/0                unassigned      YES NVRAM  administratively down down    
Loopback0                  5.5.5.5         YES NVRAM  up                    up      
R5#!
R5#show run | s ospf
R5#!
R5#show run | s bgp
router bgp 65005
 bgp log-neighbor-changes
 network 5.5.5.5 mask 255.255.255.255
 network 192.168.35.0
 network 192.168.56.0
 neighbor 1.1.1.1 remote-as 65123
 neighbor 1.1.1.1 ebgp-multihop 2
 neighbor 1.1.1.1 update-source GigabitEthernet0/0
 neighbor 2.2.2.2 remote-as 65123
 neighbor 2.2.2.2 ebgp-multihop 3
 neighbor 2.2.2.2 update-source GigabitEthernet0/0
 neighbor 192.168.35.3 remote-as 65123
 neighbor 192.168.35.3 update-source GigabitEthernet0/0
 neighbor 192.168.56.6 remote-as 65006
 neighbor 192.168.56.6 update-source GigabitEthernet0/1
R5#!
R5#show bgp sum
BGP router identifier 5.5.5.5, local AS number 65005
BGP table version is 26, main routing table version 26
12 network entries using 1728 bytes of memory
45 path entries using 3960 bytes of memory
8/5 BGP path/bestpath attribute entries using 1280 bytes of memory
6 BGP AS-PATH entries using 144 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7112 total bytes of memory
BGP activity 13/1 prefixes, 64/19 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65123      82      77       26    0    0 00:56:22       10
2.2.2.2         4        65123      83      79       26    0    0 00:58:40       11
192.168.35.3    4        65123     110      99       26    0    0 01:19:54       10
192.168.56.6    4        65006      22      23       26    0    0 00:12:50       11
R5#!
R5#show bgp
BGP table version is 26, local router ID is 5.5.5.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *    1.1.1.1/32       192.168.56.6                           0 65006 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *                     2.2.2.2                                0 65123 i
 *>                    192.168.35.3                           0 65123 i
 *    2.2.2.2/32       192.168.56.6                           0 65006 65123 i
 *                     1.1.1.1                                0 65123 i
 *                     2.2.2.2                  0             0 65123 i
 *>                    192.168.35.3                           0 65123 i
 *    3.3.3.3/32       192.168.56.6                           0 65006 65123 i
 *                     1.1.1.1                                0 65123 i
 *                     2.2.2.2                                0 65123 i
 *>                    192.168.35.3             0             0 65123 i
 *    4.4.4.4/32       192.168.56.6                           0 65006 65123 65004 i
 *                     192.168.35.3                           0 65123 65004 i
 *                     1.1.1.1                                0 65123 65004 i
 *>                    2.2.2.2                                0 65123 65004 i
 *>   5.5.5.5/32       0.0.0.0                  0         32768 i
 *    6.6.6.6/32       192.168.35.3                           0 65123 65006 i
 *                     1.1.1.1                                0 65123 65006 i
 *                     2.2.2.2                                0 65123 65006 i
 *>                    192.168.56.6             0             0 65006 i
 *    10.0.12.0/24     192.168.56.6                           0 65006 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *                     2.2.2.2                  0             0 65123 i
 *>                    192.168.35.3                           0 65123 i
 *    10.0.13.0/24     192.168.56.6                           0 65006 65123 i
 *                     1.1.1.1                  0             0 65123 i
 *                     2.2.2.2                                0 65123 i
 *>                    192.168.35.3             0             0 65123 i
 *    172.16.24.0/24   192.168.56.6                           0 65006 65123 i
 *                     192.168.35.3                           0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    2.2.2.2                  0             0 65123 i
 *    172.16.26.0/24   192.168.56.6             0             0 65006 i
 *                     192.168.35.3                           0 65123 i
 *                     1.1.1.1                                0 65123 i
 *>                    2.2.2.2                  0             0 65123 i
 *>   192.168.35.0     0.0.0.0                  0         32768 i
 *                     192.168.56.6                           0 65006 65123 i
 *                     2.2.2.2                                0 65123 i
 *                     1.1.1.1                                0 65123 i
 *                     192.168.35.3             0             0 65123 i
 *    192.168.56.0     2.2.2.2                                0 65123 65006 i
 *                     192.168.56.6             0             0 65006 i
 *>                    0.0.0.0                  0         32768 i
R5#!
R5#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
B        1.1.1.1 [20/0] via 192.168.35.3, 01:20:00
      2.0.0.0/32 is subnetted, 1 subnets
B        2.2.2.2 [20/0] via 192.168.35.3, 01:20:00
      3.0.0.0/32 is subnetted, 1 subnets
B        3.3.3.3 [20/0] via 192.168.35.3, 01:20:00
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 2.2.2.2, 00:48:18
      5.0.0.0/32 is subnetted, 1 subnets
C        5.5.5.5 is directly connected, Loopback0
      6.0.0.0/32 is subnetted, 1 subnets
B        6.6.6.6 [20/0] via 192.168.56.6, 00:12:56
      10.0.0.0/24 is subnetted, 2 subnets
B        10.0.12.0 [20/0] via 192.168.35.3, 01:20:00
B        10.0.13.0 [20/0] via 192.168.35.3, 01:13:23
      172.16.0.0/24 is subnetted, 2 subnets
B        172.16.24.0 [20/0] via 2.2.2.2, 00:43:29
B        172.16.26.0 [20/0] via 2.2.2.2, 00:09:05
      192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.35.0/24 is directly connected, GigabitEthernet0/0
L        192.168.35.5/32 is directly connected, GigabitEthernet0/0
      192.168.56.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.56.0/24 is directly connected, GigabitEthernet0/1
L        192.168.56.5/32 is directly connected, GigabitEthernet0/1
R5#!
R5#

terminal length 511
R6#!
R6#show ip int br
Interface              IP-Address      OK? Method Status                Protocol
Vlan1                  unassigned      YES NVRAM  up                    down    
GigabitEthernet0/1     unassigned      YES unset  administratively down down    
GigabitEthernet0/2     172.16.26.6     YES manual up                    up      
GigabitEthernet0/3     192.168.56.6    YES manual up                    up      
Loopback0              6.6.6.6         YES manual up                    up      
R6#!
R6#show run | s ospf
R6#!
R6#show run | s bgp
router bgp 65006
 bgp log-neighbor-changes
 network 6.6.6.6 mask 255.255.255.255
 network 172.16.26.0 mask 255.255.255.0
 network 192.168.56.0
 neighbor 172.16.26.2 remote-as 65123
 neighbor 172.16.26.2 update-source GigabitEthernet0/2
 neighbor 192.168.56.5 remote-as 65005
 neighbor 192.168.56.5 update-source GigabitEthernet0/3
R6#!
R6#show bgp sum
BGP router identifier 192.168.56.6, local AS number 65006
BGP table version is 23, main routing table version 23
12 network entries using 1632 bytes of memory
24 path entries using 1344 bytes of memory
8/5 BGP path/bestpath attribute entries using 1024 bytes of memory
6 BGP AS-PATH entries using 144 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 4144 total bytes of memory
BGP activity 12/0 prefixes, 25/1 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.16.26.2     4        65123      23      22       23    0    0 00:10:52       10
192.168.56.5    4        65005      24      23       23    0    0 00:13:31       11
R6#!
R6#show bgp
BGP table version is 23, local router ID is 192.168.56.6
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.1/32       172.16.26.2                            0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 2.2.2.2/32       172.16.26.2              0             0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 3.3.3.3/32       172.16.26.2                            0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 4.4.4.4/32       172.16.26.2                            0 65123 65004 i
*                   192.168.56.5                           0 65005 65123 65004 i
*  5.5.5.5/32       172.16.26.2                            0 65123 65005 i
*>                  192.168.56.5             0             0 65005 i
*> 6.6.6.6/32       0.0.0.0                  0         32768 i
*> 10.0.12.0/24     172.16.26.2              0             0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 10.0.13.0/24     172.16.26.2                            0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 172.16.24.0/24   172.16.26.2              0             0 65123 i
*                   192.168.56.5                           0 65005 65123 i
*> 172.16.26.0/24   0.0.0.0                  0         32768 i
*                   192.168.56.5                           0 65005 65123 i
*                   172.16.26.2              0             0 65123 i
*> 192.168.35.0     172.16.26.2                            0 65123 i
*                   192.168.56.5             0             0 65005 i
*> 192.168.56.0     0.0.0.0                  0         32768 i
*                   192.168.56.5             0             0 65005 i
R6#!
R6#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
B        1.1.1.1 [20/0] via 172.16.26.2, 00:10:56
      2.0.0.0/32 is subnetted, 1 subnets
B        2.2.2.2 [20/0] via 172.16.26.2, 00:10:56
      3.0.0.0/32 is subnetted, 1 subnets
B        3.3.3.3 [20/0] via 172.16.26.2, 00:10:56
      4.0.0.0/32 is subnetted, 1 subnets
B        4.4.4.4 [20/0] via 172.16.26.2, 00:10:56
      5.0.0.0/32 is subnetted, 1 subnets
B        5.5.5.5 [20/0] via 192.168.56.5, 00:13:35
      6.0.0.0/32 is subnetted, 1 subnets
C        6.6.6.6 is directly connected, Loopback0
      10.0.0.0/24 is subnetted, 2 subnets
B        10.0.12.0 [20/0] via 172.16.26.2, 00:10:56
B        10.0.13.0 [20/0] via 172.16.26.2, 00:10:56
      172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
B        172.16.24.0/24 [20/0] via 172.16.26.2, 00:10:56
C        172.16.26.0/24 is directly connected, GigabitEthernet0/2
L        172.16.26.6/32 is directly connected, GigabitEthernet0/2
B     192.168.35.0/24 [20/0] via 172.16.26.2, 00:10:56
      192.168.56.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.56.0/24 is directly connected, GigabitEthernet0/3
L        192.168.56.6/32 is directly connected, GigabitEthernet0/3
R6#!
R6#

Ronnie Wong

@Adam-Tyler said in Hands-on BGP lab setup:

In the case of using the next-hop-self command on R2.. Let's say you didn't want R2 to be the next hop for all routes, only 1. Is there a use case for that? Some route-map trick for that? Otherwise next-hop-self appears to be global for all BGP routes on R2 ?

Route-map is the right way to handle this. Now you have to manage multiple route-maps. So give and take.

I spent a bunch of time in the lab last night and added router 5 and 6. Everything seems to route to everything and I didn't end up having to use next-hop-self. However, not sure I would have achieved this result if I didn't fully control every router in every AS.

Correct. That is the issue with labs. We get to control everything. We know everything about every router and can manipulate every router. In reality, you will only control your AS.

So couple of follow up questions after labing all of this if you don't mind...

is an appropriate use of iBGP on the public internet in your own AS? For example, would this config be valid if all the AS numbers were ARIN issued and public IP addressing I actually owned?

You would only use iBGP peerings within your own PUBLIC AS number and not between you and another AS.

If iBGP is something you can use on the public internet, is it typical to peer every single iBGP router in our AS to the directly connected public eBGP AS routers? In this lab for example the peering between R5 and R1,R2,R3. Or do you typically only peer your internal (iBGP) routers directly connected to the ISPs? IE.. only peer R5 with R2 and R3. Or only peer R6 with R2.

You will only peer with what your ISP says you can do. You will not peer with anyone whom you're not allowed to peer to.

Next hop IP in show bgp is sometimes very confusing. take for example in my lab R5. It has an entry in show bgp for 4.4.4.4 as follows.

This is just about routing preference. Remember, BGP is designed for you to manipulate. you say where you want to route. It has to default to something. Go back and see in the BGP path preference why it chose it. Also remember that you can peer with a neighbor that is not directly connected.

Adam Tyler

@Ronnie-Wong

In the case of using the next-hop-self command on R2.. Let's say you didn't want R2 to be the next hop for all routes, only 1. Is there a use case for that? Some route-map trick for that? Otherwise next-hop-self appears to be global for all BGP routes on R2 ?

Route-map is the right way to handle this. Now you have to manage multiple route-maps. So give and take.

What would an example route-map and router bgp as-number look like to accomplish this?

for instance take this scenario. Let's say you only wanted to tell neighbor 1.1.1.1 that Router was the next hop for network 5.5.5.0/24 and NOT 6.6.6.0/24? The below block of config tells 1.1.1.1 to be the next hop for both.

Router(config)#router bgp 65123
Router(config-router)#network 5.5.5.0 mask 255.255.255.0
Router(config-router)#network 6.6.6.0 mask 255.255.255.0
Router(config-router)#neighbor 1.1.1.1 next-hop-self

Completely separate question.. Let's say you were peering with an ISP AS for the purpose of advertising public IP addresses that you own, but you weren't interested in your router loading the internet routing table. In this scenario you would just zero route everything to ISP. What BGP mechanism can be used to filter out all of those routes from coming into your BGP process, but still tell the world where your public IPs are?

Regards,
Adam Tyler

Adam Tyler

@Ronnie-Wong said in Hands-on BGP lab setup:

You would only use iBGP peerings within your own PUBLIC AS number and not between you and another AS.

Thank you. The only difference between iBGP and eBGP that I've noticed so far is simply the neighbor statement. If it uses the same AS as your current BGP process, that is iBGP. If it doesn't, that is eBGP. There has to be more to it I am missing?

Regards,
Adam Tyler

Ronnie Wong

@Adam-Tyler,

That's really the difference in configuration. this is important though. You cannot use someone else's AS number on the internet. That's bad.