Fortigate BGP cookbook of example configuration and debug commands - armandojose-devsecops/FortiGate_BGP_COOKBOOK GitHub Wiki
Welcome to the Fortigate BGP cookbook of example configuration and debug commands wiki!
BGP with two ISPs for multi-homing, each advertising a default gateway and a full routing table. Uses route-map, prefix list, weight
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee1
Prevent our Fortigate from becoming a transit AS, do not advertise learned via eBGP routes. Uses route-map, aspath-list
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee2
Force FG1 to advertise default route without having one in RIB and without using blackhole routing. Uses default-originate
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee3
Limit announced connected routes to 3.3.3.3 only. Uses route-map with redistribution
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee4
Secure BGP session between ISP1 and FG3 with one way hash. Uses MD5 authentication
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee5
Make sure we can see received routing advertisements before and after any filtering is applied. Uses soft reconfiguration
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee6
Set up BGP peering between FG3 and FG1 using loopback in FG3
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee7
Remotely Triggered Black Hole Routing configuration
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/#ee8
The BGP configuration flow in general is:
List of all useful BGP debug and verification commands:
show router bgp
get router info bgp summary
get router info bgp network
get router info routing-table bgp
get router info bgp neighbors
get router info bgp neighbors advertised-routes
get router info bgp neighbors routes
get router info bgp neighbors received-routes
diagnose sys tcpsock | grep 179
diagnose ip router bgp level info
diagnose ip router bgp all enable
exec router clear bgp all
BGP with two ISPs for multi-homing, each advertising default gateway and full routing table
Task: Configure 2 BGP peerings with different providers, each ISP advertising to us (FG3, AS 1680) both, default and Internet routes. Limit the learned routes from each ISP to default route only. Advertise to both ISPs our internal network of 10.10.10.1, making sure clients on the Internet prefer ISP1 (AS 111) to reach this network. Also we want to use ISP1 to reach the Internet, and only if it fails to use ISP2.
Solution:
The topology of this case:
Fortigate BGP case 1 network topology diagram
FG3, AS1680:
- Create prefix list to allow ONLY default route (0.0.0.0/0) and deny everything else.
config router prefix-list edit "accept-dflt-only" config rule edit 1 set prefix 0.0.0.0 0.0.0.0 unset ge unset le next end next end
* Prefer ISP1 to reach the Internet, having ISP2 as backup in case of failure. The easiest way to do so is via weight setting, which can be used inside config neighbor to set the weight for ALL routes learned from this neighbor. Or it can be used by first config route prefix-list to match specific route(s), then setting the weight for these specific matched routes inside config router route-map, which in turn will be applied to the neighbor. The other way would be to increase Local Preference of the routes learned from ISP1, but this would require to configure route-map, an additional extra step.
Here we are not trying to prefer specific routes via ISP1 but all routes learned from it, so I will set weight on the neighbor.
* The next step is to make sure my advertised route 10.10.10.1 is reachable via both ISPs, but is preferred by Internet clients via ISP1. Usually you do it by prepending your own AS number to the advertised route(s). I create route-map to do so:
config router route-map edit "prepend-out" config rule edit 1 set set-aspath "1680 1680" next end next end
**Now I can configure both BGP peers on FG3, including redistributing the connected networks (here it is 10.10.10.1/32 of the loopback interface) to BGP: **
config router bgp set as 1680 config neighbor edit "12.12.12.12" set prefix-list-in "accept-dflt-only" set remote-as 111 set weight 10 next edit "13.13.13.6" set prefix-list-in "accept-dflt-only" set remote-as 222 set route-map-out "prepend-out" next end config redistribute "connected" set status enable end
Verification.
As remote peers are not configured yet, the status will be oscillating between Active and Connect:
get router info bgp summary
FG3-AS1680 # get router info bgp summary BGP router identifier 10.10.10.1, local AS number 1680 BGP table version is 1 1 BGP AS-PATH entries 0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
12.12.12.12 4 111 0 0 0 0 0 never Active
13.13.13.6 4 222 0 0 0 0 0 never Active
...
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
12.12.12.12 4 111 0 0 0 0 0 never Active
13.13.13.6 4 222 0 0 0 0 0 never Connect
FG1, FG6.
BGP settings of both peers are almost identical (except local to each AS number and FG3 peering IP) so I will list just FG1. One note: unlike in FG3, which distributes into BGP the directly connected loopback 10.10.10.1, I need both Fortigates here to advertise default route 0.0.0.0/0 which they don't have. As I mentioned in the Configuration Flow graph - BGP will only advertise routes present in the active routing table (RIB) by default. The Fortigate has 2 ways to circumvent this BGP standard requirement: we can announce the default route with capability-default-originate, and for other routes we can use set network-import-check disable. But I am not using either of them here. To satisfy this condition, I add blackhole route to the 0.0.0.0/0 route, in Cisco world it is called "route to Null0". This adds 0.0.0.0/0 as static route which I can redistribute into BGP. Note 1: Additionally, to simulate "Internet" IPs, I added 8.8.8.8 as loopback in both FG1 and FG6 and redistribute them via redistribute connected.
Note 2: Important point I glossed over in FG3 is router-id. Fortigate (as well as Cisco and most others) will take the highest IP address on the loopback interface available unless explicitly set. In this specific setup I have 8.8.8.8 address on both FG1 and FG6 set on their loopbacks to advertise them as "Internet" addresses to FG3. And this may cause a problem - if any BGP peer detects its own router-id coming from the peer, the BGP session will be torn down with NOTIFICATION sent. So, here it is a must, but generally is a good idea to set router-id manually to unique IP address. I will add unique router-id to FG3 and FG6.
When such situation of duplicate router-id happens, Fortigate will show the error:
BGP: 12.12.12.12-Outgoing [DECODE] Open: Invalid Router ID 8.8.8.8
FG1:
config router static edit 1 set dst 0.0.0.0/0 set blackhole enable next end
Verification
Note: to save me typing, I add this alias to show routing table:
config system alias edit "rt" set command "get router info routing all" next end
So when you see it in the output instead of the full command get router info routing all know it is an alias, and not a secret hidden command in Fortigate :).
alias rt
Routing table for VRF=0 Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default
S* 0.0.0.0/0 [10/0] is a summary, Null <--- This is the default route we want to be present in the RIB and now it is. C 2.2.2.2/32 is directly connected, Loop1 C 8.8.8.8/32 is directly connected, Loop2 C 13.13.13.0/24 is directly connected, port1
Now let's configure the BGP on FG1:
config router bgp set as 111 set router-id 1.1.1.1 config neighbor edit "12.12.12.3" set remote-as 1680 next end config redistribute "connected" set status enable config redistribute "static" set status enable end
Verification
First, let's see if the BGP peering with two ISPs has been established (yes, it has). On FG3:
FG3-AS1680 # get router info bgp summary
BGP router identifier 10.10.10.1, local AS number 1680 BGP table version is 7 3 BGP AS-PATH entries 0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 12.12.12.12 4 111 126 297 5 0 0 00:02:35 1 13.13.13.6 4 222 121 288 6 0 0 00:02:12 1
Total number of neighbors 2
Let's see if we are getting default route from both peers:
****get router info bgp network 0.0.0.0/0
FG3-AS1680 # get router info bgp network 0.0.0.0/0 BGP routing table entry for 0.0.0.0/0 Paths: (2 available, best #2, table Default-IP-Routing-Table) <--- Yes, we do Advertised to non peer-group peers: 13.13.13.6 <--- This is not good, read further why 222 13.13.13.6 from 13.13.13.6 (6.6.6.6) Origin incomplete metric 0, localpref 100, valid, external Last update: Wed May 20 12:06:00 2020
111 12.12.12.12 from 12.12.12.12 (1.1.1.1) <--- default route from ISP1 Origin incomplete metric 0, localpref 100, weight 10, valid, external, best <--- preferred because its weight is 10 Last update: Wed May 20 12:05:58 2020 <--- the 2nd ISP peer has weight not set, think 0
Now we need to make sure we advertise our network 10.10.10.1 to both peers:
This is what we advertise to ISP1:
****FG3-AS1680 #get router info bgp neighbors 12.12.12.12 advertised-route
BGP table version is 3, local router ID is 10.10.10.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight RouteTag Path *> 3.3.3.3/32 12.12.12.3 32768 0 ? *> 10.10.10.1/32 12.12.12.3 32768 0 ? *> 12.12.12.0/24 12.12.12.3 32768 0 ? *> 13.13.13.0/24 12.12.12.3 32768 0 ?
Total number of prefixes 4
Looks good - we advertise 10.10.10.1, as well as other directly connected networks from port1, port2, and loopback.
And what do we advertise to the ISP2?
FG3-AS1680 # get router info bgp neighbors 13.13.13.6 advertised-routes
BGP table version is 3, local router ID is 10.10.10.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight RouteTag Path *> 0.0.0.0/0 13.13.13.3 10 0 1680 1680 111 ? *> 3.3.3.3/32 13.13.13.3 32768 0 1680 1680 ? <--- route-map prepends twice AS 1680 *> 10.10.10.1/32 13.13.13.3 32768 0 1680 1680 ? *> 12.12.12.0/24 13.13.13.3 32768 0 1680 1680 ? *> 13.13.13.0/24 13.13.13.3 32768 0 1680 1680 ?
Total number of prefixes 5
As you probably noticed there are too many routes! Indeed we have a problem here - instead of advertising just our (AS 1680) routes as we do to the ISP1, we advertise also the routes we learned from ISP1 (0.0.0.0/0)! We have become transit AS - if ISP2 does not filter incoming from us routes, their clients may potentially reach networks behind ISP1 via us, and for free. Also ISP1 may see our re-advertisements of their routes as BGP hijacking, and be very unhappy about that. To fix this issue we need to implement route filtering, be it on ISP1 & ISP2 sides, or on our outgoing advertisements. I will look into it in the next scenario.
The good news is that the route-map prepending AS 1680 to make ISP2 less preferred for our network works.
Let's have a look at the work the prefix-list filtering is doing on FG3. The BGP debug should show it: FG3-AS1680 # diagnose ip router bgp level info
Here I set BGP debug level to INFO, as the default level of ERROR will not show enough information. Next I can run the debug. diagnose ip router bgp all enable
Unfortunately as the BGP session is already established nothing really happens, so I clear ALL BGP sessions (not something you want to do on production Fortigate lightly):
exec router clear bgp all
BGP: 13.13.13.6-Outgoing [FSM] State: OpenConfirm Event: 26 id=20300 logdesc="BGP neighbor status changed" msg="BGP: %BGP-5-ADJCHANGE: neighbor 13.13.13.6 Up " <--- The BGP session with ISP2 is established BGP: 13.13.13.6-Outgoing [DECODE] Update: NLRI Len(15) BGP: 13.13.13.6-Outgoing [FSM] State: Established Event: 27 BGP: 13.13.13.6-Outgoing [RIB] Update: Received Prefix 0.0.0.0/0 <--- And here we can see prefix-list filtering BGP: 13.13.13.6-Outgoing [RIB] Update: Prefix 13.13.13.0/24 denied due to filter <--- in action , 0.0.0.0/0 is accepted but BGP: 13.13.13.6-Outgoing [RIB] Update: Prefix 8.8.8.8/32 denied due to filter <--- the rest of received routes are discarded BGP: 13.13.13.6-Outgoing [RIB] Update: Prefix 2.2.2.2/32 denied due to filter
Disable all debug: diagnose debug reset
Prevent our Fortigate from becoming a transit AS, do not advertise learned via eBGP routes.
As seen in the previous case, without any filtering on FG3 everything it learns from its BGP peers and is being installed in its routing table will be advertised to all the BGP peers.
We can prevent it in few ways:
- Filter outgoing advertisements to include only our networks by IPs (not very scalable, but granular)
- Filter outgoing advertisements using AS number (much more scalable, but not granular)
First is to explicitly allow our own networks in outgoing advertisements and block everything else. About blocking everything else - both prefix lists and ACLs have implicit deny any any, so it is not necessary to explicitly deny everything else. Matching networks using prefix lists
Prefix lists use Prefix (network) and the Prefix Length (subnet mask length in bits) to look at when comparing the routes. Some examples of using prefix lists:
The following prefix-list will allow just networks 10.10.10.1/32 and 3.3.3.3/32:
config router prefix-list edit "own-nets-only-out" config rule edit 1 set prefix 10.10.10.1 255.255.255.255 unset ge unset le next edit 2 set prefix 3.3.3.3 255.255.255.255 unset ge unset le next end next
What is left is to apply the prefix list outbound to both peers on FG3:
config router bgp set as 1680 config neighbor edit "12.12.12.12" set prefix-list-in "accept-dflt-only" set prefix-list-out "own-nets-only-out" set remote-as 111 set weight 10 next edit "13.13.13.6" set prefix-list-in "accept-dflt-only" set prefix-list-out "own-nets-only-out" set remote-as 222 set route-map-out "prepend-out" next end
The advertised routes before: In this example , get router info bgp neighbors 13.13.13.6 advertised-routes
and get router info bgp neighbors 12.12.12.12 advertised-routes
And after applying the prefix list:
get router info bgp neighbors 13.13.13.6 advertised-routes
and
get router info bgp neighbors 12.12.12.12 advertised-routes
****Matching networks using AS PATH list
BGP advertised routes bear with them quite a lot of information on which we can match/filter and do other manipulation to our liking. One of the Well-known mandatory (i.e. present in EVERY route advertisement/withdrawal in BGP, the others being ORIGIN, and NEXT_HOP) is the AS_PATH attribute. So we can use it to allow advertising only our own routes with AS PATH lists. AS path lists use regular expressions to match the AS numbers in the path. The regex differs slightly from the familiar PCRE/sed. Special symbols understood:
Let's look at some examples of matching AS numbers.
Now back to our FG3, let's create and apply AS path list filtering to advertise only our own nets to the BGP peers.
****Step 1. Create aspath-list matching local routes:
config router aspath-list
edit "LocalRoutesOnly"
config rule
edit 1
set action permit
set regexp "^$"
next
end
next
end
Step 2. Create if needed (for ISP1) and/or edit existing route-map (for ISP2 there is already prepend-out for prepending AS) that uses the aspath-list for matching.
config router route-map edit "prepend-out" config rule edit 1 set match-as-path "LocalRoutesOnly" <-- adding the match for local routes only set set-aspath "1680 1680" next end next
edit "LocalRoutesOut"
config rule
edit 1
set match-as-path "LocalRoutesOnly"
next
end
next
end
Step 3. Finally, apply route-map LocalRoutesOut to ISP1 and refreshing BGP session with ISP2 to activate the changes. NOTE: You do NOT need to reset/clear/soft clear BGP session in Fortigate after changing filtering of any kind (route-maps, prefix-lists, as-paths) for them to take effect. All modern versions of FortiGate have route-refresh capability turned on by default, so any change in filtering takes effect 2-3 minutes after being configured. Just wait a bit.
config router bgp set as 1680 config neighbor edit "12.12.12.12" set prefix-list-in "accept-dflt-only" set remote-as 111 set route-map-out "LocalRoutesOut" <-- I removed here prefix-list ""own-nets-only-out" " as unnecessary anymore set weight 10 next edit "13.13.13.6" set prefix-list-in "accept-dflt-only" set remote-as 222 set route-map-out "prepend-out" next end config redistribute "connected" set status enable end
Verification
ISP1, routes received from FG3: in this case get router info bgp neighbor 12.12.12.3 routes
Based on this reference page:
https://yurisk.info/2020/05/20/fortigate-bgp-cookbook-of-example-configuration-and-debug/