Hot!VXLAN over IPSEC

Author
jfgagnon@synovatec.com
New Member
  • Total Posts : 17
  • Scores: 2
  • Reward points: 0
  • Joined: 2018/04/06 09:41:35
  • Status: offline
2018/04/13 08:37:53 (permalink)
0

VXLAN over IPSEC

Hi!
 
I was wondering if any of you could helpo me out making this work,
I"m runnning 2 VM64 Fortigate on a ESXi server, through 2 VyOS router to emulate.
Version 5.6.3
The tunnel is up, but somehow, ARP requests are not getting through:
 

FortiGate-VM64 # diag netlink brctl name host VXLAN-INTERFACE
show bridge control interface VXLAN-INTERFACE host.
fdb: size=2048, used=3, num=3, depth=1
Bridge VXLAN-INTERFACE host table
port no device devname mac addr ttl attributes
1 6 port4 00:0c:29:d6:62:ab 51 Hit(51)
2 17 VXLAN 5e:9f:e8:0f:21:a6 0 Local Static
1 6 port4 00:0c:29:0f:47:91 0 Local Static
 
 
 

interfaces=[any]
filters=[host 10.0.11.100 and arp]
15.470412 port4 in arp who-has 10.0.11.101 tell 10.0.11.100
15.470449 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100
16.487104 port4 in arp who-has 10.0.11.101 tell 10.0.11.100
16.487121 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100
17.511047 port4 in arp who-has 10.0.11.101 tell 10.0.11.100
17.511059 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100
18.535167 port4 in arp who-has 10.0.11.101 tell 10.0.11.100
18.535191 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100
 
 
 
 
here's my config:
 
edit "port2"
set vdom "root"
set ip 84.84.85.2 255.255.255.0
set allowaccess ping
set type physical
set alias "WAN1"
set role wan
set snmp-index 2
next
 

edit "VXLAN"
set vdom "root"
set type tunnel
set snmp-index 12
set interface "port2"
next
 

config vpn ipsec phase1-interface
edit "VXLAN"
set interface "port2"
set peertype any
set proposal des-md5
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 84.84.85.2
set encap-remote-gw4 84.84.86.2
set remote-gw 84.84.86.2
set psksecret ENC OWif8UtnjVfxFQDRN8ajAv/Ten/+O8xoWmIRA1fylLgeGljO1jb+irdNGhDpwlOJD5SJzW4uycM4fDZ2ISwWZUzCCeGKS2q2Df8PQ+qz4Q3pKS4FRd1/IpIYC1dcnnpsEixK5NuYyThTKHc9AoCZF0FT3akcZjevsHKb9m+CV/6VNE9ZY6mDy9bwcDrc7mSiie+mIg==
next
end
 

config vpn ipsec phase2-interface
edit "VXLAN_ph2"
set phase1name "VXLAN"
set proposal des-md5
next
end
 

config system switch-interface
edit "VXLAN-INTERFACE"
set vdom "root"
set member "port4" "VXLAN"
set intra-switch-policy explicit
next
end
 

config firewall policy
edit 1
set name "VXLAN-INCOMING"
set uuid 1d96cbcc-3d91-51e8-585d-00de8ce55269
set srcintf "VXLAN"
set dstintf "port4"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set nat enable
next
edit 2
set name "VXLAN-OUTGOING"
set uuid 2c5fe85a-3d91-51e8-7c00-653d11fab724
set srcintf "port4"
set dstintf "VXLAN"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set nat enable
next
end
 
 
 
Thanks for the help!
#1

13 Replies Related Threads

    jfgagnon@synovatec.com
    New Member
    • Total Posts : 17
    • Scores: 2
    • Reward points: 0
    • Joined: 2018/04/06 09:41:35
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 06:46:09 (permalink)
    0
    No one?
    #2
    emnoc
    Expert Member
    • Total Posts : 5108
    • Scores: 318
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 06:59:46 (permalink)
    0
    The cli-cmd diag debug flow would be useful here but when you dump oin either end of the ipsec do you see ARPs?
     
    Also what does diag vpn tunnel list shows for any counters ( tx/rcv-enc )
     
    BTW: your configurations looks right
     
    ken
     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #3
    jfgagnon@synovatec.com
    New Member
    • Total Posts : 17
    • Scores: 2
    • Reward points: 0
    • Joined: 2018/04/06 09:41:35
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 10:34:20 (permalink)
    0
    I don't see ARP getting inside the tunnel
     
     
    FortiGate-VM64 # diag sniffer packet any 'host 10.0.11.101 and arp' 4
    interfaces=[any]
    filters=[host 10.0.11.101 and arp]
    0.925592 port4 in arp who-has 10.0.11.100 tell 10.0.11.101
    0.925610 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101
    1.949396 port4 in arp who-has 10.0.11.100 tell 10.0.11.101
    1.949411 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101
    2.973408 port4 in arp who-has 10.0.11.100 tell 10.0.11.101
    2.973430 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101
    3.997561 port4 in arp who-has 10.0.11.100 tell 10.0.11.101
    3.997577 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101
    #4
    HA
    Gold Member
    • Total Posts : 149
    • Scores: 6
    • Reward points: 0
    • Joined: 2010/09/19 07:10:45
    • Location: Luxembourg
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 10:49:44 (permalink)
    0
    Hi,
     
    Maybe something missing on the physical interface to forward broadcast ??
    Here's my config...
     
    config system interface
    edit "wan1"
    set ip 10.0.0.1 255.255.255.0
    next
    edit "wan2"
    set vlanforward enable
    set broadcast-foward enable
    set l2forward enable
    set stpforward enable
    set netbios-forward enable
    next
    edit "VxLan-IPsec"
    set vlanforward enable
    set broadcast-foward enable
    set l2forward enable
    set stpforward enable
    set netbios-forward enable
    next
    end

    config system switch-interface
    edit "VxLan-Switch"
    set member "wan2" "VxLan-IPsec"
    set intra-switch-policy explicit
    next
    end
     
    Hope it can help you...
     
    Regards,
     
    HA
    #5
    emnoc
    Expert Member
    • Total Posts : 5108
    • Scores: 318
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 12:16:15 (permalink)
    0

    I don't see ARP getting inside the tunnel

     
     
    the VXLAN is name is your tunnel,you need to execute  the same on the opposite side but again you need to lok at vpn tunnel statistics and diag debug flow
     
    You should see the messages for the action of vxlan or something similar .
     
    ken
     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #6
    jfgagnon@synovatec.com
    New Member
    • Total Posts : 17
    • Scores: 2
    • Reward points: 0
    • Joined: 2018/04/06 09:41:35
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 12:52:18 (permalink)
    0
    Any commands I could use then?
    Cause what I got not very useful so far
    #7
    emnoc
    Expert Member
    • Total Posts : 5108
    • Scores: 318
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/17 13:01:21 (permalink)
    0
    i would  start with cli-cmd
     
    diag vpn tunne list name VXLAN
     
     
    Execute the  same on both  sides of the ipsec-gw, do you see tunnel counters for ENC-domain  tx/recv pkts ? if not  than you have a problem
     
     
    e.g
     
    VPN1  sent 3400 pkts  than VPn2 should have  3400
     
    if  VPN1 received  200 pkts than VPN2 should  have sent 2000 pkts
     
     
    for diag debug flow, run the search here for  examples  but again cli cmd
     
    diag debug en
    diag debug reset
    diag debug flow filter addr x.x.x.x
    diag debug flow show console enable
     
    diag debug flow trace start 100
     
     
    ( now generate some traffic for the ENC_DOMAINS and see what happens )
     
    Ken
     
     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #8
    jfgagnon@synovatec.com
    New Member
    • Total Posts : 17
    • Scores: 2
    • Reward points: 0
    • Joined: 2018/04/06 09:41:35
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/18 08:03:56 (permalink)
    0

    FortiGate-VM64 # diag debug flow show console enable
    command parse error before 'console'
    Command fail. Return code -61
     
    #9
    jfgagnon@synovatec.com
    New Member
    • Total Posts : 17
    • Scores: 2
    • Reward points: 0
    • Joined: 2018/04/06 09:41:35
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/18 08:10:53 (permalink)
    0
    It does look like traffic is not getting through, i only see bytes going up in incoming on one side, and outgoing on the other side
    #10
    emnoc
    Expert Member
    • Total Posts : 5108
    • Scores: 318
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: VXLAN over IPSEC 2018/04/18 09:38:43 (permalink)
    0
    So your closer to your  identification of the problems(s)
     
    Have you ran the diag debug flow and validate the configurations are correct?
     
    Ken
     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #11
    jan_vanek
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/05/08 23:33:03
    • Status: offline
    Re: VXLAN over IPSEC 2018/05/09 00:06:53 (permalink)
    0
    Hi everybody,
    I think it is not necessary to create a new thread so I will kindly join this one. I try to setup VXLAN over IPSEC according to these guides:
    http://kb.fortinet.com/kb/documentLink.do?popup=true&externalID=FD40170
    https://travelingpacket.com/2017/09/28/fortigate-vxlan-encapsulation/
    My goal is to  build a L2 VPN between two FortiGates using VxLAN over IPsec. Or simply put: interconnect a detached branch to HQ according this scenario:
    CISCO switchport --- (trunk)----HQ_Fortigate ----Internet(IPSEC)----Branch_ Fortigate---(trunk)---Cisco switchport
     
    I hope I am close to be successfull however I am still fighting (probably) with MTU/MSS settings. The maximum ICMP packet size between clients on HQ and Branch side (incl. header)  is 1390 Bytes regardless to "Don't Fragment Flag is set or not. Larger packet does not pass. That of course causes many problems with http/https traffic and other "usual" network services. 
     
    Could anyone advise how to handle with MTU/MSS settings on Fortigate? Firmware is v5.6.4. I can provide all the necessary output form debug.
     
    Thank you in addition
    #12
    Selective
    Expert Member
    • Total Posts : 2740
    • Scores: 115
    • Reward points: 0
    • Joined: 2007/07/03 10:44:56
    • Location: Gothenburg - Sweden
    • Status: offline
    Re: VXLAN over IPSEC 2018/05/09 04:54:42 (permalink)
    0
    Hi Jan,
     
    There is no fix, only a workaround for the MTU issue:
     
  • FortiOS does not send back an ICMP “destination unreachable, fragmentation needed and DF set” to the source when an IP packet with the DF bit set and a size greater than the tunnel MTU cannot be forwarded inside the VxLAN-IPsec tunnel.The workaround is to stop honoring the DF bit:
     
    config system global
        set honor-df disable
    end
     
    Source: http://kb.fortinet.com/kb/microsites/search.do?cmd=displayKC&docType=kc&externalId=FD40170&sliceId=1&docTypeID=DT_KCARTICLE_1_1&dialogID=34936883&stateId=1%200%2034938616
     
  • #13
    jan_vanek
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/05/08 23:33:03
    • Status: offline
    Re: VXLAN over IPSEC 2018/05/10 23:58:06 (permalink)
    0
    Hi 
    Thank you for your feedback and very inspirational topic. I've done some investigation and here is my temporary conclusion. 
    set honor-df disable possibly can do certain job in case I want to forward untagged traffic (switchport in access mode). However this setting does not affect tagged traffic (switchport in trunk mode). I've noticed that it is better to use separate VDOM for software switch and corresponding interfaces because of arp request. That may help to the founder of this topic. Untaged traffic would be let say good enough but I wanted more.
     
    Here is my solution:
    1)I've created an usual IPSEC tunnel with loopback interfaces. Notice, there in no "set encapsulation vxlan"
     
    config vpn ipsec phase1-interface
    edit "vxlan_ph1"
    set interface "wan1"
    set ike-version 2
    set peertype any
    set proposal aes256-sha512
    set nattraversal disable
    set remote-gw remote_IP_address
    set psksecret ENC mysecredpassword
    next
    end
    config vpn ipsec phase2-interface
    edit "vxlan_ph2"
    set phase1name "vxlan_ph1"
    set proposal aes256-sha512
    set auto-negotiate enable
    set src-subnet 172.30.31.0 255.255.255.0   //IP address of local loobpack inteface. (Of cource, there could be /32 prefix :-) )
    set dst-subnet 172.30.30.0 255.255.255.0  //IP address of remote loobpack inteface. 
    next
    end
     
    2) Next step is to configure "native" vxlan according this reference: http://help.fortinet.com/fos50hlp/56/Content/FortiOS/fortigate-whats-new/Top-Network-vxlan.htm
     
    config system vxlan
    edit "vxlan1"
    set interface "loop1"  //local loopback interface
    set vni 1
    set remote-ip "172.30.30.1"  //remote loopback int. IP address
    next
    end
     
    This configuration makes the "vxlan1" interface:
     
    [font="'courier new', courier; font-size: small"]edit "vxlan1"
    set vdom "root"
    set type vxlan
    set snmp-index 9
    set interface "loop1"
     
    3) The last step is to put the physical port "internal4" and vxlan interface "vxlan1" into the soft switch:
     
    config system switch-interface
    edit "sw_switch"
    set vdom "root"
    set member "internal4" "vxlan1"  //
    next
    end
     
    After that I am able to transfer tagged traffic from HQ switchport to the branch switchport without packet loss and MTU is no problem any more. Unfortunatelly, I have to break next investigation / debug because I had to return the borrowed device. Anyway, I hope that my approach could help.
     
    *English is not my mother tongue, please excuse any errors on my part.
    Jan
     
    #14
    Jump to:
    © 2019 APG vNext Commercial Version 5.5