Hot!FG60E - FG60E IPSec Tunnel Slow

Page: 12 > Showing page 1 of 2
Author
remosito
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/02/08 03:31:17
  • Status: offline
2017/04/27 07:38:31 (permalink) 5.4
0

FG60E - FG60E IPSec Tunnel Slow

Hi everybody,
 
our Site2Site IPSec VPN Tunnel between two FTG-60E on 5.4.4 is slow and I have run out of stuff to try.
 
Both Sites are connected via Gbps Fiber Internet. Iperf3 to a public Iperf Server gave 600+ Mbps results on both ends. So we know our Internet uplinks are good. But our IPSec Tunnel tops out at 130-ish Mbps.
 
I have tried from tips I read in various places:
 - deleted the udp-dns session helper thingie
 - took all internal1-8 out of the internal hardware switch
 - tried different enc algos
 - set local-gw
 
 
here the sanitized relevant parts of the config from MainSite FTG-60E
 

config system npu
    set enc-offload-antireplay enable
    set offload-ipsec-host enable
end

config system interface
    edit "wan1"
        set vdom "root"
        set ip a.a.a.a 255.255.255.248
        set allowaccess ping ssh
        set type physical
        set estimated-upstream-bandwidth 1000000
        set estimated-downstream-bandwidth 1000000
        set role wan
        set snmp-index 1
        set secondary-IP enable
        config secondaryip
            edit 1
                set ip a.a.a.b 255.255.255.255
            next
        end
    next


    edit "internal1"
        set vdom "root"
        set ip p.p.p.p 255.255.255.0
        set allowaccess ping https ssh
        set type physical
        set snmp-index 11
    next

config vpn ipsec phase1-interface
    edit "VPN-BRN-LSN"
        set interface "wan1"
        set local-gw a.a.a.a
        set peertype any
        set comments "VPN: VPN-BRN-LSN (Created by VPN wizard)"
        set remote-gw f.f.f.f
        set psksecret ENC secretsauce
    next
end

config vpn ipsec phase2-interface
   edit "VPN-BRN-LSN"
        set phase1name "VPN-BRN-LSN"
        set proposal aes256-sha256
        set dhgrp 14
        set comments "VPN: VPN-BRN-LSN (Created by VPN wizard)"
        set src-subnet p.p.p.p 255.255.255.0
        set dst-subnet q.q.q.q 255.255.255.0
    next


 
and here some diag output from the same 60E:
 

diagnose vpn ipsec status
All ipsec crypto devices in use:
NP6LITE_0
        null: 0 0
        des: 0 0
        3des: 0 0
        aes: 1682560 1894400
        aes-gcm: 0 0
        aria: 0 0
        seed: 0 0
        null: 0 0
        md5: 0 0
        sha1: 6784 8192
        sha256: 1675776 1886208
        sha384: 0 0
        sha512: 0 0
NPU HARDWARE
        null: 0 0
        des: 0 0
        3des: 0 0
        aes: 33331 0
        aes-gcm: 0 0
        aria: 0 0
        seed: 0 0
        null: 0 0
        md5: 0 0
        sha1: 544 0
        sha256: 32754 0
        sha384: 0 0
        sha512: 0 0
SOC3:
        null: 0 0
        des: 0 0
        3des: 0 0
        aes: 20 32
        aes-gcm: 0 0
        aria: 0 0
        seed: 0 0
        null: 0 0
        md5: 0 0
        sha1: 1 2
        sha256: 18 30
        sha384: 0 0
        sha512: 0 0
SOFTWARE:
        null: 0 0
        des: 0 0
        3des: 0 0
        aes: 0 0
        aes-gcm: 0 0
        aria: 0 0
        seed: 0 0
        null: 0 0
        md5: 0 0
        sha1: 0 0
        sha256: 0 0
        sha384: 0 0
        sha512: 0 0

get system npu
enc-offload-antireplay: enable
dec-offload-antireplay: enable
offload-ipsec-host : enable


diagnose vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=VPN-BRN-LSN ver=1 serial=3 a.a.a.a:0->f.f.f.f:0
bound_if=3 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/8 options[0008]=npu
proxyid_num=1 child_num=0 refcnt=20 ilast=13 olast=13 auto-discovery=0
stat: rxp=1715632 txp=1911092 rxb=833611779 txb=1426710396
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=1101
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=VPN-BRN-LSN proto=0 sa=1 ref=5 serial=1
  src: 0:p.p.p.p/255.255.255.0:0
  dst: 0:q.q.q.q/255.255.255.0:0
  SA: ref=6 options=2e type=00 soft=0 mtu=1438 expire=11928/0B replaywin=1024 seqno=29811 esn=0 replaywin_lastseq=00025e80
  life: type=01 bytes=0/0 timeout=43174/43200
  dec: spi=257f4fb4 esp=aes key=32 secretsauce
       ah=sha256 key=32 secretsauce
  enc: spi=8574257e esp=aes key=32 secretsauce
       ah=sha256 key=32 secretsauce
  dec:pkts/bytes=155266/49170368, enc:pkts/bytes=173054/97413586
  npu_flag=03 npu_rgwy=f.f.f.f npu_lgwy=a.a.a.a npu_selid=0 dec_npuid=1 enc_npuid=1


 
kinda all looks as it should be to me.
 
I can post the same info for the other FGT-60E if needed. Or any other output or config parts....
 
any help greatly appreciated
#1

20 Replies Related Threads

    remosito
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/02/08 03:31:17
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/04/28 06:19:27 (permalink)
    0
    No takers?
     
    Would be really glad for any ideas or feedback or throughput values of others with two 60E over gigabit.
     
    60E has a new processor that is supposed to be twice as fast in IPSec as the 90D (2Gbps vs 1Gbps). And for the 60D I had found somebody achieving 600-800 Mbps ( https://blog.webernetz.net/2016/03/15/fortigate-vpn-speedtests/ )...
     
     
    #2
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/05/05 16:19:01 (permalink)
    0
    I have a 1 gbps link too with slow IPSEC VPN.
     
    FG200D 5.2.10 to FG60D 5.2.11.
    500 to 700mbps in one direction and 124 mbps in the other. I will replace my FG60D with a FG60E for testing purposes if I don't find a solution soon.
     

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #3
    remosito
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/02/08 03:31:17
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/05/08 00:49:02 (permalink)
    0
    Paul S
    I have a 1 gbps link too with slow IPSEC VPN.
     
    FG200D 5.2.10 to FG60D 5.2.11.
    500 to 700mbps in one direction and 124 mbps in the other. I will replace my FG60D with a FG60E for testing purposes if I don't find a solution soon.
     




    124mbps is pretty much the speeds I get with my two 60Es. Coincidence or unpublished limitation by the 60 line?
     
    Would be most interested in knowing what speeds you get with a 60E. But if I were you haven't that 60E lying around, but need to buy it. I think I would not risk it and go 90...
    #4
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/05/08 16:38:14 (permalink)
    0
    I have a 60E lying around so I will try that and see what happens.
     
    I've upgraded to 5.4.4 on my 60D and I get over 124 mbps now, but never over around 700+ for up+down. I want to get 700mbps+ in both directions for 1.4gbps total. The wireless link that this fiber replaced maxed out around 700 mbps per direction, so I was hoping for at least that.
     
     
    post edited by Paul S - 2017/08/15 15:48:55

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #5
    Smartypants
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2016/02/29 10:29:48
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/05/31 10:58:45 (permalink)
    0
    I wish I could get 130Mbps.
    I have a 900D HA Pair trying to connect to a VM version of the Fortigate FW over a 500MBps circuit on my end and a bigger pipe at the remote data center and I'm only getting 30-50Mbps. Both me and the Data Center engineering team are at a loss.
    Opened a ticket with support (we pay a lot $$ for support) and was told to buy IPerf and run tests and report back.
    So much fro support ;-)
    I'm stuck at this point we planned on using Forti's for a site 2 site and we can't and I don't know why.
    We have ASA's on a different network and they work so maybe I need to do that but I don't really want to do that.
     
    #6
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/06/01 10:06:05 (permalink)
    0
    iperf is a free utility. What OS versions?  does your CPU usage get high on either side? are you using encryption and hashing algorithms supported by ASIC?  Since Fortigate-VM has no ASIC you will probably have a limit due to CPU speed, but I image it is much higher than you are seeing now.

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #7
    ispcolohost
    Silver Member
    • Total Posts : 66
    • Scores: -1
    • Reward points: 0
    • Joined: 2014/11/18 08:06:51
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/07/14 14:05:47 (permalink)
    0
    Hey all, I'd be interested in any updates you might have on the throughput issue.  I have not been able to find out if Fortinet's newer SoC3 hardware actually accelerates SHA256, or by how much.  I know the hardware without an NPU, and up through the NP4, tunnels using SHA256 would cause all VPN traffic to hit the CPU where SHA1 lets it stay on the NPU.  Based on their 'hardware acceleration' document:
     
    http://docs.fortinet.com/uploaded/files/2855/fortigate-hardware-acceleration-54.pdf
     
    I can tell that NP6/NP6Lite should accelerate an AES256/SHA256 tunnel, but of course there's zero guidance on what the real world throughput should be and their support seems unwilling to release an estimate for anything.
     
    I've been running into horrid throughput between 200D's with AES256/SHA256/DH14 tunnels, as well as between 200D's and remote 60D and 100D locations.  I came across this thread while trying to research what real world expectations I should have on the 100E with SoC3 (same as 60E/80E in theory but no knowledge of clock speed), and the 200E with CP8+NP6Lite, as we have some remote locations that would get a 60E or 100E instead of 200E.
     
    The data sheets are of course useless since they just say IPSec with no mention of encryption or hash algorithms and that seems to make or break Fortigate hardware.
     
     
     
    #8
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/15 15:56:24 (permalink)
    0
    After doing some troubleshooting today with my ISP regarding our 1 Gbps circuit running at 100 Mbps, I am contributing the following information:
     
    Setup:
    - IPSEC VPN (AES-256/SHA1) between FWF60D (5.4.4) and FG200D (5.2.10).
    - All interfaces at 1 Gb, full Duplex.
    - with policy set to "set auto-asic-offload disable"
    iperf test shows only around 91 mbps throughput.
     
    after running command: "unset auto-asic-offload" on the policy. NO other changes.
     
    iperf test shows 650 to 700 mbps, in one direction. in+out maxes out at around ~725mbps.
     
    conclusions:
    - enabling ASIC does in fact improve VPN performance significantly.
    - without ASIC I can only get 100 Mbps through my IPSEC VPN.
    - ASIC (NP4Lite) will accelerate SHA256/SHA1
    - There is another limit somewhere in the 60D or the 200D that prevents me from getting closer to 1 Gbps in each direction.
     
    remaining questions I have not answered yet:
    - Would the NP6Lite ASIC on the 60E get higher throughput on the IPSEC VPN?
    - Would throughput change on the 60D or the 60E if I used AES256/SHA256?

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #9
    ispcolohost
    Silver Member
    • Total Posts : 66
    • Scores: -1
    • Reward points: 0
    • Joined: 2014/11/18 08:06:51
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/15 16:05:54 (permalink)
    0
    I have some E-series units on the way and support & sales both claim it offloads SHA256, so we shall see.
    #10
    ecsupport
    New Member
    • Total Posts : 17
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/02/19 18:38:33
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/16 00:04:44 (permalink)
    0
    Throwing my 50 cents in ;)

    Got 2x 50Es 5.4.4 (one just upped to 5.4.5) with a tunnel up, 1Gbps fiber one end and a 120x25 cable on the other end. Best I can do upload on fiber is 45Mbps.

    Now Here's the crazy part- I tried an SSLVPN from a laptop on the gig fiber side hitting the cable 50E and I was getting the full 125M up! This is well above the 100M limit on the 50E PDF for SSLVPN! And I don't know the real limit because I was hitting the remote circuits download limit.

    Tried all the usual- MTU, disable enable offload, various SHA & DH & AES combos no luck. Support is shooting in the dark.
    #11
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/16 15:24:28 (permalink)
    0
    Does the 50E have ASIC? How high is the 50E CPU usage during your 45 Mbps?

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #12
    ecsupport
    New Member
    • Total Posts : 17
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/02/19 18:38:33
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/16 15:35:18 (permalink)
    0
    The 50E does have an ASIC but I don't think it has the full hardware enc-dec capabilities of the 60 and later, since its only rated for 200Mbps IPSEC throughput vs 1Gbps (60D/E) per the flawed datasheets.
     
    CPU is low during transfers, maybe 5-10%. If offload is disabled then it jumps to about 40-45%.
    #13
    bgp179
    New Member
    • Total Posts : 8
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/04/23 08:59:37
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/25 07:08:09 (permalink)
    0
    Hey all,
     
    Jumping into this thread.  We have exact setup as OP here with two 60E's over a 1Gbps private p2p line.  When encryption is enabled (offload disabled because of Win 2012/ECN issue) we get about 100 Mbps tops.  That's with AES256/SHA1 on phase1 and AES256/SHA256 on phase 2.   But when we take the traffic off the tunnel and just route between the two 60E's, we get near 800 Mbps.  Significant.
     
    We are going to upgrade to 5.4.5 and enable the ASIC offloading, hopefully the 2012 ECN works.  But maybe we'll also move phase2 to AES256/SHA1 as it seems that is supported for offloading.
     
    Thoughts?
    #14
    ispcolohost
    Silver Member
    • Total Posts : 66
    • Scores: -1
    • Reward points: 0
    • Joined: 2014/11/18 08:06:51
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/25 08:51:43 (permalink)
    0
    That's going to be very disappointing if the 60E doesn't offload SHA256.  Our VPN throughput is abysmal because of SHA2 between all our D-series stuff.  We've  received a bunch of 100E/200E equipment for various larger sites, but had hoped to use the 60E for small sites to resolve this issue.  Have not yet deployed any of it...
    #15
    fl0at0xff
    Bronze Member
    • Total Posts : 38
    • Scores: 0
    • Reward points: 0
    • Joined: 2016/08/23 00:13:56
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/08/26 00:10:41 (permalink)
    0
    Have you tried to use less secure proposals for the phase 2 ? 
    #16
    bgp179
    New Member
    • Total Posts : 8
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/04/23 08:59:37
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2017/09/07 07:38:17 (permalink)
    0
    So the upgrade to 5.4.5 fixed both the Windows 2012 ECN issue and we are getting good throughput with AES256/SHA256 with NPU offloading.
     
    Can't believe that worked haha.
     
     
    #17
    Toshi Esumi
    Expert Member
    • Total Posts : 1251
    • Scores: 89
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2018/02/20 15:30:47 (permalink)
    0
    I want to dig up this case since we got a set of demo 60Es and put in iperf test-bed in our lab:
    test-serverA<->FG60E(wan1)<- IPsec ->(wan1)FWF60E<->test-serverB
    And if we broke the internal (hard-switch) interface to single internalx interfaces, TCP test shows 800-900Mbps since that's about the max performance of ServerA<->ServerB. But if we don't break the hard-switch, we got 300-350Mbps.
    Does anyone got a similar test result?
    #18
    Paul S
    Gold Member
    • Total Posts : 164
    • Scores: 8
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2018/02/21 12:25:49 (permalink)
    0
    what OS version? per my previous post I was getting around 700 Mbps with iperf.

    FG200D 5.6.5 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.6.5  |  Fortimail 5.3.11
    Network+, Security+
    #19
    Toshi Esumi
    Expert Member
    • Total Posts : 1251
    • Scores: 89
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: FG60E - FG60E IPSec Tunnel Slow 2018/02/21 12:42:10 (permalink)
    0
    Sorry I forgot that important info. Both FG/FWF60E are running 5.4.8.
    #20
    Page: 12 > Showing page 1 of 2
    Jump to:
    © 2018 APG vNext Commercial Version 5.5