SD WAN with dual IPsec and DialUp Branch

[Jirka1]

Hi to all,

I plan to do the following topology:

HQ: FGT81E with 2 WAN (Static Public IP) BR1: 60E with 1 WAN (DialUp) BR1: 60E with 1WAN (DialUp) 2 IPsec tunnels will be built from BR to HQ (wan1 and wan2).

Trying to accomplish the following :: https://kb.fortinet.com/k....do?externalID=FD41297 with the difference that I have two BR. Internet from the BR must be passed through the HQ (NAT and UTM will be applied).

So, is this solvable in dial up connections? Can dialup interface be a member of SD wan? Or what other way do you recommend?

Thank you for the constructive idea.

All boxes have 6.2.2

Jirka

[Jirka1]

ok so i tried it and it seems that once ipsec dial up tunnels are added to sd wan and are configured as 0.0.0.0/->0.0.0.0/0, DR routes to these tunnels will appear in the Route Monitor 0.0.0.0->ipsec_tnl1, 0.0.0.0->ipsec_tnl2 - which I think is wrong. So I'll try to use specific branch subnets in Phase 2 and we'll see. Jirka

[Toshi_Esumi]

The interface type shouldn't be a matter for SD-WAN as long as the FGT can put the automatic default-route toward all member interfaces.

But I have a doubt with that KB setup because the Branch's internet trafic is configured to go out "to_port1_p1" to go through HQ but return route is coming through "port2_p1" under normal circumstances. I didn't think SD-WAN wouldn't make exception for "asymmetric-route" block. But I might be wrong. So far I haven't found statement either way. Please let me know if it actually works.

[Jirka1]

Hi Toshi, so after a few hours of exploring, debugging and coffee I think I have done - except one important thing: LAN traffic from HQ and self-originating traffic from HQ FGT does not work. I mean that I cannot access the Internet from the HQ FGT LAN ports and CLI. For me, for unknown reason, traffic from HQ LAN ports is randomly routed to all IPsec tunnels instead of WAN1 or WAN2:

 id=20085 trace_id=237 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=1, 192.168.2.1:34897->8.8.8.8:2048) from local. type=8, code=0, id=34897, seq=0."
id=20085 trace_id=237 func=init_ip_session_common line=5666 msg="allocate a new session-0001827b"
id=20085 trace_id=237 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-LEV_wan2"
id=20085 trace_id=237 func=ipsecdev_hard_start_xmit line=842 msg="Failed to find IPsec Common: LEV_wan2"
id=20085 trace_id=238 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=1, 192.168.2.1:34897->8.8.8.8:2048) from local. type=8, code=0, id=34897, seq=1."
id=20085 trace_id=238 func=resolve_ip_tuple_fast line=5581 msg="Find an existing session, id-0001827b, original direction"
id=20085 trace_id=238 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-LEV_wan2"

id=20085 trace_id=238 func=ipsecdev_hard_start_xmit line=842 msg="Failed to find IPsec Common: LEV_wan2"

LEV_wan2 is a second IPsec tunnel to the branch :\ 

SD wan rules I have correct, correct order, I tried to define the input interface (set input-device <interface>) see: https://docs.fortinet.com.../cookbook/848980/self-originating-traffic all without success.

It is also strange that even the self-originating traffic from HQ does not work. I tried to enter src-addresses in WAN1 and WAN2 configuration:

(virtual-wan-link) # config members

(members) # show
config members
    edit 5
        set interface "LEV_wan1"
    next
    edit 6
        set interface "LEV_wan2"
    next
    edit 4
        set interface "wan1"
        set gateway 62.209.xxx.xxx
    next
    edit 7
        set interface "wan2"
        set gateway 178.17.xxx.xxx
    next
    edit 8
        set interface "PHA_wan1"
    next
    edit 9
        set interface "PHA_wan2"
    next
end

(members) # edit 4
(4) # set source <intreface-ip>

Nothing, nothing, nothing :\

According to the routing table, are DR routers not only to wan1 and wan2, but also to IPsec tunnels for branch offices? Why? Because they are members of SD WAN?

# get router info routing-table all

Routing table for VRF=0
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S* 0.0.0.0/0 [1/0] via 10.10.1.2, PHA_wan1
                  [1/0] via 10.10.2.2, PHA_wan2
                  [1/0] via 10.11.1.2, LEV_wan1
                  [1/0] via 10.11.2.2, LEV_wan2
                  [1/0] via 62.209.xxx.xxx, wan2
                  [1/0] via 178.17.xxx.xxx, wan1
C 10.10.1.1/32 is directly connected, PHA_wan1
C 10.10.1.2/32 is directly connected, PHA_wan1
C 10.10.2.1/32 is directly connected, PHA_wan2
C 10.10.2.2/32 is directly connected, PHA_wan2
C 10.11.1.1/32 is directly connected, LEV_wan1
C 10.11.1.2/32 is directly connected, LEV_wan1
C 10.11.2.1/32 is directly connected, LEV_wan2
C 10.11.2.2/32 is directly connected, LEV_wan2
C 178.17.xxx.xxx/27 is directly connected, wan2
C 62.209.xxx.xxx/26 is directly connected, wan1
C 192.168.2.0/24 is directly connected, port1

I really do not understand why SD WAN rules will not be reflected in WAN1 and WAN2. Even if I put them at the top ... Jirka

[Jirka1]

here is a listing of all SD wan rules - I see no reason why traffic to the Internet is sent to IPsec tunnels. I tried also upgrade to 6.2.3, downgrade to 6.2.1..

 # diagnose firewall proute list
list route policy info(vf=root):

id=2131689475 vwl_service=3(PHA_to_LEV) vwl_mbr_seq=6 5 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=28 oif=27
source(2): 192.168.200.0-192.168.200.255 172.16.98.0-172.16.98.255
destination(2): 192.168.202.0-192.168.202.255 192.168.100.0-192.168.100.255

id=2131689476 vwl_service=4(LEV_to_PHA) vwl_mbr_seq=8 9 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=29 oif=30
source(2): 192.168.202.0-192.168.202.255 192.168.100.0-192.168.100.255
destination(2): 192.168.200.0-192.168.200.255 172.16.98.0-172.16.98.255

id=2131689473 vwl_service=1(SSL_to_PHA) vwl_mbr_seq=8 9 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=29 oif=30
users and groups(1): VPN_users(2)
source(1): 10.212.134.200-10.212.134.210
destination(4):
        10.10.2.0-10.10.2.3
        10.10.1.0-10.10.1.3
        192.168.200.0-192.168.200.255
        172.16.98.0-172.16.98.255


id=2131689474 vwl_service=2(SSL_to_LEV) vwl_mbr_seq=6 5 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=28 oif=27
users and groups(1): VPN_users(2)
source(1): 10.212.134.200-10.212.134.210
destination(4):
        192.168.202.0-192.168.202.255
        192.168.100.0-192.168.100.255
        10.11.2.0-10.11.2.3
        10.11.1.0-10.11.1.3


id=2131689478 vwl_service=6(IPsec_to_WAN) vwl_mbr_seq=7 4 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=6 oif=5
source(4):
        10.10.2.0-10.10.2.3
        10.10.1.0-10.10.1.3
        10.11.2.0-10.11.2.3
        10.11.1.0-10.11.1.3

destination(1): 0.0.0.0-255.255.255.255

id=2131689477 vwl_service=5(ALL_to_WAN) vwl_mbr_seq=7 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=6
source(1): 0.0.0.0-255.255.255.255
destination(1): 0.0.0.0-255.255.255.255

I am starting to feel that it is better to leave SD wan only for WAN1 and WAN2 and set OSPF for IPsec. Jirka

[Toshi_Esumi]

Only things I can tell now is below:

- There is only one "SD-WAN" instance is available. So if you want to control traffic (priority/failover/etc.) between interfaces, all - regardless wan interfaces or IPSec interfaces - need to be a member of SD-WAN.

- Once they're put in SD-WAN as members, all of them need to have a default route so that FGT can steer specific traffic you configured in the rules(service in CLI) using the policy-routes (or its mechanism). This is the exact same mechanism working with multiple static default routes + regular policy routes toward the internet before SD-WAN was introduced.

I'm still learning SD-WAN myself. It's probably not so easy for someone online can point out/guess what's exactly going on, or not going on, which might change dynamically. I would suggest you get help from FTNT TAC.

[Jirka1]

toshiesumi wrote:

Only things I can tell now is below:

- There is only one "SD-WAN" instance is available. So if you want to control traffic (priority/failover/etc.) between interfaces, all - regardless wan interfaces or IPSec interfaces - need to be a member of SD-WAN.

- Once they're put in SD-WAN as members, all of them need to have a default route so that FGT can steer specific traffic you configured in the rules(service in CLI) using the policy-routes (or its mechanism). This is the exact same mechanism working with multiple static default routes + regular policy routes toward the internet before SD-WAN was introduced.

I'm still learning SD-WAN myself. It's probably not so easy for someone online can point out/guess what's exactly going on, or not going on, which might change dynamically. I would suggest you get help from FTNT TAC.

Yes, I agree. But in this case, I really don't understand why FGT doesn't accept the order of SD-WAN rules. Now I set up a support ticket.

Merry X-mass

Jirka

[Toshi_Esumi]

That's what I'm looking for as well. Please update how the ticket goes for the rest of us.

[Jirka1]

Hi Toshi, FTN support solved it as follows: Since self-originating traffic in version 6.2.2 and higher does not pass SD WAN - https://kb.fortinet.com/k....do?externalID=FD47380 - it is necessary to change DR. But..in all KB and CB from Fortinet, it is stated that when using SD WAN, only one DR per SD WAN is required - which is obviously not always true...

So if there are some IPsec tunnels in SD WAN that connect local ranges (eg HQ and BR), it is necessary to place these IP local address ranges into Static Routes and set DR to the net (WAN1 and WAN2) directly to the gateway of the upstream router.

I had no other explanation from them. But the ticket says: Bug fix already available. From this I understand that it is probably a bug and will be fixed. But who knows .. Anyway, now everything seems to work correctly (I will try it in detail tomorrow).

This is how my routing table looks like now (FTN support used all RFC1918 address):

Jirka

[Toshi_Esumi]

To me the issue before 6.2.2 described in the KB sounds like if you have BGP peering on wan1 and wan2 with ISPs, BGP messages for ISP1 from wan1 IP can be steered to wan2 depending of the SD-WAN decision at that time. 6.2.2 or later doesn't have the problem because those self-oritinated traffic is excluded from SD-WAN rules' (policy routes) steering.

The routing table you showed seems to be HQ's. Then TAC removed internet paths from SD-WAN (at least for static routes) and left only VPNs to Branch in. So those private ranges needed to have different static routes toward SD-WAN(VPN pair). That sounds like a bug and that's probably what TAC mentioned about with the bug fix.

.