Hot!Bad bufferbloat on WAN link. How to shape with Fortigate

Author
richg
New Member
  • Total Posts : 4
  • Scores: 0
  • Reward points: 0
  • Joined: 2018/07/16 01:15:47
  • Status: offline
2018/07/16 01:38:47 (permalink)
0

Bad bufferbloat on WAN link. How to shape with Fortigate

Hi all,
 
Have recently started a new contracting gig. part of the role is implementing a voip telephone system , and I've been investigating the network a little as there are some problems with jitter and large latency spikes to handsets. Anecdotally users are also reporting "slow" internet , often when we are no where near peak capacity. 
 
a (not managed by us) telco router/media convertor is onsite (either one or both , I see a cisco MAC from the fortigate WAN interface, its near the MDF in the building, which we don't have access to) .  with a 50/50 fibre link.
RRUL testing shows pretty bad bufferbloat 
 
 
 

 
I'm not very familiar with fortigate products, I don't see any option for fq_codel , HTB etc as  such , which I have had some success  implementing on linux based routers etc before. 
 
Im thinking much of this problem is either because of how the ISP internet gear is buffering traffic (if its a router I can see in ARP), or its just discarding everything above 50m. I see spikes over 50mbit when the link is saturated that drop off quickly, I dont think they are letting us burst traffic though, I think its just being dropped so I need to setup some shaping outbound. 
 
There is pretty much zero setup on the fortigate right now from the outfit that installed it. No QoS. There are Vlans but they do nothing except have slightly different subnets (all route to each other,  no tagging or QoS). There are stacked DELL switches attached to the LAN, everything in the office goes through these.
 
Anyone have some experience trying to solve this on fortigate gear , or some tips on config?
 
in the past ive worked with mid band ethernet type services where its fairly essential to shape traffic before handing it off to the NTU ( a dumb layer 2 device thats just mirroring the mac from the switch in the exchance). I'm thinking if I can just shape everything at the LAN interface to slightly less than 50 this will improve, then I can work on QoS for the voice vlan etc. 
 
 
 
Any ideas or tips? I think we can get much better performance from this service.  
 
 
#1

13 Replies Related Threads

    pireality
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/02 03:01:03
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 03:03:59 (permalink)
    0
    This is an excellent question, I wonder why nobody has responded in over 3 weeks.  I would like the same assistance.
    #2
    rwpatterson
    Expert Member
    • Total Posts : 8359
    • Scores: 197
    • Reward points: 0
    • Joined: 2006/08/08 10:08:18
    • Location: Long Island, New York, USA
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 04:56:12 (permalink)
    0
    Welcome to the forums guys.
     
    There are CLI options to tell the Fortigate the bandwidth that you are subscribed to. (Both inbound and outbound. On the interface in the cli, type 'set?' and see the list of available options.) That along with proper policy shaping should quell those traffic drops and hopefully help get all you traffic through during periods of high traffic.

    -Bob - self proclaimed posting junkie!
    See my Fortigate related scripts at: http://fortigate.camerabob.com

    -4.3.19-b0694
    FWF60B
    FWF80CM (4)
    FWF81CM (2)
     
    #3
    pireality
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/02 03:01:03
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 05:41:28 (permalink)
    0
    I have tried setting the bandwidth on the interfaces and it doesn't change the speed in my testing at all.  I just used the gui, under interfaces.  I continually get 120mbps down and about 12mbps up, even if I set the interface bandwidth to 50mbps and 5mpbs I still get 120/12?  It is like it isn't working.  
    Additionally, do you have a good link on the traffic shaping piece that I could read in order to get it setup correctly?
     
    Thanks for your speedy response BTW!
    #4
    rwpatterson
    Expert Member
    • Total Posts : 8359
    • Scores: 197
    • Reward points: 0
    • Joined: 2006/08/08 10:08:18
    • Location: Long Island, New York, USA
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 06:01:39 (permalink)
    0
    One caution: The speed may be in mega BYTES. Look for a small or capital B.
     
    Divide the speed you thought you have by 8 and see if that improves things.
    post edited by rwpatterson - 2018/08/02 06:03:34

    -Bob - self proclaimed posting junkie!
    See my Fortigate related scripts at: http://fortigate.camerabob.com

    -4.3.19-b0694
    FWF60B
    FWF80CM (4)
    FWF81CM (2)
     
    #5
    pireality
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/02 03:01:03
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 06:09:08 (permalink)
    0
    I did some testing with the DF bit set and found that packets > 1472 fail, so I set the WAN interface max mtu to 1472, however, this hasn't had any effect, like the other changes to the traffic rate on the interface.
     
    Thanks for the answers, keep em coming!
    #6
    pireality
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/02 03:01:03
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 06:12:48 (permalink)
    0
    Oh yeah, one more thing, http://evenroute.com/iqrouter/ seems to have a dynamic adjustment for line speed and apparently removes bufferbloat "automagically".
     
    I would assume if I get the configuration you are suggesting working, I won't need that device, but I am so tired of laggy/slow connections, I'd really like to fixor one way or the other.
     
    Thanks!
    #7
    pireality
    New Member
    • Total Posts : 11
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/02 03:01:03
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/02 07:20:15 (permalink)
    0
    I confirmed mbps, oddly, the max mtu size changed from this morning's 1472 to now it is 1444....why would that change dynamically during the day?  Weird.
    #8
    crispy
    Bronze Member
    • Total Posts : 24
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/11/05 04:34:04
    • Location: Australia
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/08/05 18:42:21 (permalink)
    0
    Hi All,
     
    On the Fortigate there is two options on the cli for setting the bandwith. The option that you can set from the GUI sets the parameter of "estimated-upstream-bandwidth" and "estimated-downstream-bandwidth" which is only used to estimate the links utilisation. There is another option which you can only set from the cli called "inbandwidth" and "outbandwidth" which is the bandwidth limit to apply to the interface.
     
    I use these settings in association with dscp and the traffic shapers on the fortigates to provide QoS for phone and video when required and it works well. If you have not already, download the cli guide from the docs.fortinet.com site as it will explain a lot of these cli only commands in more detail.
     
    crispy
    #9
    richg
    New Member
    • Total Posts : 4
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/07/16 01:15:47
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2018/09/25 21:28:00 (permalink)
    0
    Just an update/query , I haven't had time to revisit this for a while. 
     
    Still have the latency under load issues, I'm now suspecting its could be way too large buffer sizes on the layer 3 switch stack. I don't see a way to fix this easily unfortunately. I've qos'd the voice vlan now which has helped a little,  but I still get extensions lagging (up to 2000ms) occasionally and it doesn't correlate with WAN load.  Theres plenty of room left on the WAN link , and latency is still just appalling at times. Its less related to load and more to the types of traffic (big flows seem to mess with latency) as far as I can see. 
     
    This isn't really a QoS or shaping issue. This is a buffering issue that manifests well before the WAN link is saturated. 
     
    I'm going to try and experiment on the fortigate by manually tweaking txquelen on the WAN interface during some planned works coming up afterhours. It could have some unforeseen consequences so I don't recommend trying this without a console cable handy or during business hours ;) It also may not work on your particular model (we have an 80E here). I've verified this works quickly and doesn't crash the firewall , but I need to test it. Your mileage may vary etc. 
     
    you have to execute ifconfig using fnsysctl to manually set these values.eg:
     
    fnsysctl ifconfig wan1 txquelen 100 
     
     
     It's current set to 1000 (just run fnsysctl ifconfig to see what its set to) , which is probably very excessive for a smallish WAN link. I can verify setting the interface bandwidth has no bearing on this value.  I'll try it on the wan link and the ethernet back into the switches. 
     
    I'll report back if this makes any difference whatsoever to my flent tests.  
     
    My question - are fortigate planning to implement any sort of SQM in the future? linux / BSD /  mac os , openwrt, pfsense, palo alto and many others have all implemented something like fq_codel.... ;)
     
    This would probably go a long way to resolving problems like this... 
    #10
    wallky
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2019/02/17 14:00:37
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2019/02/17 14:04:40 (permalink)
    0
    Was there any outcome on this? I have a very similar if not exactly the same issue:
     
    One ISP here in Australia uses Cisco as their NTU and we plug Fortigate's into the Cisco. The link is a 50/50Mbps, and we get 50Mbps down, but up is only getting 25Mbps.

    I have an egress shaper applied to the WAN interface, have set all of the estimated speeds etc but it makes no difference.
     
    I've set the egress shaper to 10Mbps and it only gets 10Mbps. I set it to 20Mbps and it only gets 20Mbps, but anything above 25 just stops at between 25-30Mbps.
     
    It only happens on this one ISP, as I have many Foritgates out there with other ISP's and we don't have this issue.
    They are saying it's our CPE.
     
    I'm interested to see if changing the txquelen made any difference to your problem, as I suspect it's something similar to my issue.
     
    Thanks!
    #11
    richg
    New Member
    • Total Posts : 4
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/07/16 01:15:47
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2019/02/18 21:10:27 (permalink)
    0
    txque change minimal impact.
    shaping outbound to a bit less than full speed and lowering that made a slight difference.
     
    in the end the solution was to put a netgate (pfsense) firewal in and setup fq_codel
     
    voip latency is around 25ms now solid , even under load.
     
    this is in australia too. its a telstra service resold by someone else, I suspect they are doing something screwy in the interconnect because its not the only one I've seen now from the same ISP. 
    #12
    wallky
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2019/02/17 14:00:37
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2019/02/19 04:13:21 (permalink)
    0
    Thanks. I've got the same issue with a 60E, 100E and 200E on Telstra fibre connections with 50Mbps and 200Mbps, some direct with Telstra and others resold by another ISP with the ISP's data.

    They have all told me that it's to do with the way Telstra uses Cisco routers as their NTU and how is discards excess traffic above the speed you are paying for.
     
    I've got plenty of Fortigate's out there with Vocus and TPG fibre and are not having this issue on the upload side.
     
    I've tried egress traffic shaping via GUI and also CLI via interface shaping and using percentage shaping, but can't get this to resolve.
     
    The ISP has sent me a Juniper router which I've put in place of the Fortigate and they have put interface shaping on and it gets the full 50Mbps of the link, so they are blaming the Fortigate.
     
    I don't want to have to put another router or device in front of the Fortigate. 

    So any other ideas?
     
    richg
    txque change minimal impact.
    shaping outbound to a bit less than full speed and lowering that made a slight difference.
     
    in the end the solution was to put a netgate (pfsense) firewal in and setup fq_codel
     
    voip latency is around 25ms now solid , even under load.
     
    this is in australia too. its a telstra service resold by someone else, I suspect they are doing something screwy in the interconnect because its not the only one I've seen now from the same ISP. 




    #13
    richg
    New Member
    • Total Posts : 4
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/07/16 01:15:47
    • Status: offline
    Re: Bad bufferbloat on WAN link. How to shape with Fortigate 2019/02/19 15:26:05 (permalink)
    0
    I don't know if that's the same issue exactly, that was clearly bufferbloat. latency seesawed stupidly after around 40% load. 
     
    try running a RRUL test with flent just to be sure. you will need a MAC/linux box (or a VM and good NIC in your computer) and a netperf/iperf server.
     
    what sits behind the fortigate? I also found some issues in the dell switch stack they had there as well. it was cabled wrong for a start. this was also causing problems. 
     
    where abouts are you speed testing from? directly off the fortigate?
     
    have the telco given you any indication why they believe it's CPE, besides the Juniper getting full speed? I'd be curious to know how they tested that exactly.. 
     
    what do your traffic reports show (telco ones) ?
    how do those line up with the fortigate reports?
    do you have netflow/sflow setup ? 
     
    You could try hard shaping upstream to link speed minus a delta (try 10%) and see if that makes a difference.
     
    also make sure you have set the link speed correctly , from memory there are two places to do it, ones effectively a label in the GUI , but the other is set via cli like this:
     
    config system interface
    edit wan1
    set inbandwidth <kb>
    set outbandwidth <kb>
     
    ..may vary depending on your fortios version. 
     
    if it is a buffer issue then there are unfortunately a lot of places it can happen. NIC's (OS buffer and hardware buffers) , your switches , the CPE, the telcos gear at other end. etc. Its often telcos, because everyones got obsessed with not dropping packets, which is counterproductive. if you are maxxing something you want packets to drop so TCP will backoff. 
     
    I know its not particulary helpful, but swapping the fortigate for a pfsense box improved the situation significantly. 
     
    I really hope fortigate follows some of the other major vendors in implementing some form of SQM at some stage.. but I'm not sure that this is the same problem.
     
    from my telco days certain types of handoff require hard policing on customer end, usually layer 2 ones. else packets get dropped in access network .
    #14
    Jump to:
    © 2019 APG vNext Commercial Version 5.5