BGP From Fortigate To Cisco not changing routing table

[The_Nude_Deer]

From the rough diagram below , I have an issue with SDWAN spoke, it has 2 IPSEC tunnels as its SDWAN members, one to DC1 where the LAN sits behind a Cisco Switch and another IPSEC to DC2 same setup.

The 2 DC switches swap routes via EIGRP, the Hub Fortigates in each DC , have EBGP to the switches where the routes have been re-distibuted from EIGRP, this is an inherited set up which does need to change!

When I "break" the IPSEC to DC1, By disabling the IPSEC interface, traffic correctly now goes down the SDWAN via the DC2 ipsec , I can see this happening no problem, The issue is the Cisco Switch does not see this at all, and still believes the LAN is behind its EBGP neighbour in DC1?

From DC2 Switch, the route to the LAN on the spoke 192.168.1.0/24 should now be via DC2 Fortigate connected to its Switch

* 192.168.1.0/24 > DC2-FORTIGATE 0 65400 ?
*> DC1-SWITCH 25600512 32768 ?

However, The DC2 SWITCH shows the above, the route is there with a weight of 0 , and its picked the best route via DC1, its no longer reachable, so why is the route there?

I then up the interface, and the routing tables stay the same on the switches, i have to clear ip bgp on the switches to make it all work again,

Hope you guys can offer some help thats giving me sleepless nights.

[srajeswaran]

 I feel the top device is Hub and the DC firewalls are spokes, I just simplified the topology as below, please correct me if I missed the logic.

In simple words, when VPN between Hub and DC1 firewall is down, the SW2 route should point to DC2 FW an not SW1? Is this correct?

If I understand correctly the Switch is choosing EIGRP route over EBGP route, is this correct? If so, isnt this an issue on the Switch ?

[The_Nude_Deer]

Hi Suraj, no as per my drawing the top device is a spoke (There are more of them, I am just using 1 in the drawing to make it easier), that has 2 IPSEC tunnels to each DC. The top device (spoke) is in IBGP 65400 with DC1 FW and DC2 FW, DC1 Path is prefered in the SDWAN configuration (Manual selection)

When DC1FW ipsec tunnel is gone (I manually disable it on the spoke as a test) the SDWAN spoke, correctly used the DC2 FW Tunnel to send traffic down. However, the routes on SW1 still believe the network behind the spoke is is via its DC1 FW,  When I check the DC1FW advertised routes, it is still advertising the SPOKE subnet? how can it, if its down! this is what the problem is.. I have tried weight, local pref, adjusted BGP timers, have also added BFD but that never sees the BFD neighbour. Appreciate you answering, I have had 2 weeks of trying to figure this out!

[srajeswaran]

Thanks for the clarification. Can you share the below outputs from DC1FW after disabling the tunnel?

get router info routing-table 192.168.1.0/24

get router info bgp network

Can you also confirm "network-import-check" is enabled? ref : https://community.fortinet.com/t5/FortiGate/Technical-Tip-Advertise-a-BGP-route-not-present-in-the-r...

[The_Nude_Deer]

network-import-check is enabled yes,

this command : get router info routing-table 192.168.1.0/24 is not recognised,

get router info bgp network as it currently is on FG DC1: (10.230.0.8 is the tunnel interface on the spoke)

*>i192.168.1.0/24 10.230.0.8 0 100 0 0 ? <-/1>

routing on the cisco switches are currently as I would expect:

DC1-SWITCH#show ip route | i 192.168.1.0
B 192.168.1.0/24 [20/0] via 10.99.2.82, 13:29:03 (10.99.2.82 is the Fortigate)

so BGP correctly pointing at the spoke subnet

DC2-SWITCH#show ip route | i 192.168.1.0
D EX 192.168.1.0/24 [170/25600512] via 10.99.2.5, 13:30:34, Vlan150 (10.99.2.5 is the interface on FHQ Switch)

so EIGRP correctly pointing DC2 towards DC1 Switch then up to the Fortigate and onto the spoke.

Ill post the results now, when it breaks

[The_Nude_Deer]

With the Tunnel down (disabled) 

* i192.168.1.0/24 10.231.0.8 0 100 0 0 ? <-/-> (10.231.0.8 is the tunnel interface to the secondary DC)
*>i 10.231.0.8 0 100 0 0 ? <-/1>
* i 10.231.0.8 0 100 0 0 ? <-/->

The COMMAND on DC1-FW:

get router info bgp neighbors 10.99.2.81 advertised-routes

*> 192.168.1.0/24 10.99.2.82 100 0 0 ? <-/->

DC1 FG is still advertising the spoke subnet as the best and valid route ?

With the Tunnel down (disabled) 

* i192.168.1.0/24 10.231.0.8 0 100 0 0 ? <-/-> (10.231.0.8 is the tunnel interface to the secondary DC)
*>i 10.231.0.8 0 100 0 0 ? <-/1>
* i 10.231.0.8 0 100 0 0 ? <-/->

The COMMAND on DC2-FW:

get router info bgp neighbors 10.99.2.57 advertised-routes

*> 192.168.1.0/24 10.99.2.58 100 0 0 65400 65400 ? <-/->

Also advertsing! 

[srajeswaran]

Correction "get router info routing-table details 192.168.1.0/24".

Can you get this output from the DC1FW and DC2FW when the VPN is down on DC1FW.

[The_Nude_Deer]

#DC1FORTIGATE# get router info routing-table details 192.168.1.0/24

Routing table for VRF=0
Routing entry for 192.168.1.0/24
Known via "bgp", distance 200, metric 0, best
Last update 00:00:53 ago
* 10.231.0.8 (recursive via 10.99.2.81, INSIDE-NETWORK)  has it learned this from DC1 switch??

DC2FORTIGATE # get router info routing-table details 192.168.1.0/24

Routing table for VRF=0
Routing entry for 192.168.1.0/24
Known via "bgp", distance 200, metric 0, best
Last update 14:39:00 ago
* 10.231.0.8 (recursive is directly connected, DC2-SPOKE-VPN)

THE DC1 Switch, still sees the spoke route via its FG
*> 192.168.1.0/24 10.99.2.82 0 65400 ? 

[srajeswaran]

This shows it is still learning the route. 10.99.2.81 is the SW2?

#DC1FORTIGATE# get router info routing-table details 192.168.1.0/24

Routing table for VRF=0
Routing entry for 192.168.1.0/24
Known via "bgp", distance 200, metric 0, best
Last update 00:00:53 ago
* 10.231.0.8 (recursive via 10.99.2.81, INSIDE-NETWORK

[The_Nude_Deer]

My comments above:

* 10.231.0.8 (recursive via 10.99.2.81, INSIDE-NETWORK)  has it learned this from DC1 switch??

its the DC1 Switch its leaning it from, perhaps this is to do with the EIGRP into BGP on the switch? something you cant help with right ? :(