Internet Drop Outs after Fortinet Install

[Ckny]

After Fortinet Firewall install we experienced Internet and network dropouts and got a report showing horrible dropouts, latency, lost packets, through the roof. What could cause? Do I need higher bandwidth from my ISP? A report showed the ISP may be cutting me off due to increased bandwidth use. I never had the problem with the other firewalls. How can the Fortinet be generating that much more bandwidth? All other hardware, configurations and applications have not changed. Only the Firewall.

[Dave_Hall]

Without further details about the fgt model/firmware/ISP (WAN) connection, brief network layout, I going to say check the duplex/speed on the WAN connect to the ISP's gateway/modem device.  eg. from the CLI enter diagnose hardware deviceinfo nic <WAN interface)

The output should look similar to this:

# diagnose hardware deviceinfo nic wan1 Description         Intel(R) Gigabit Ethernet Network Driver Driver_Name         igb Driver_Version      5.0.6 PCI_Vendor          8086 PCI_Device_ID       1533 PCI_Subsystem_Vendor          ffff PCI_Revision_ID     0003 PCI_Bus             5 PCI_Slot            0 MAC_Type            6 PCI_Bus_Type        PCI-E PCI_Bus_Speed       2.5Gb/s PCI_Bus_Width       Width x1 IRQ                 16 System_Device_Name  wan1 Current_HWaddr      90:6c:ac:3e:dx:xx Permanent_HWaddr    90:6c:ac:3e:dy:yy Link                up Speed               1000 Duplex              full FlowControl         current:0/requested:3 Interrupt mode      MSI-X Rx queue(s)         1 Tx queue(s)         1 Rx_Packets          42064231 Tx_Packets          32484440 Rx_Bytes            50613981859 Tx_Bytes            4808163593 Rx_Errors           0 Tx_Errors           0 Rx_Dropped          0 Tx_Dropped          0 Multicast           445470 Collisions          0 Rx_Length_Errors    0 Rx_Over_Errors      0 Rx_CRC_Errors       0 Rx_Frame_Errors     0 Rx_FIFO_Errors      0 Rx_Missed_Errors    0 Tx_Aborted_Errors   0 Tx_Carrier_Errors   0 Tx_FIFO_Errors      0 Tx_Heartbeat_Errors 0 Tx_Window_Errors    0 Tx_Single_Collision_Frames    0 Tx_Multiple_Collision_Frames  0 Tx_Deferred         0 Rx_Frame_Too_Longs  0 Rx_Frame_Too_Shorts 0 Rx_Align_Errors     0 Rx_Flow_Control_XON 0 Rx_Flow_Control_XOFF          0 Tx_Flow_Control_XON 0 Tx_Flow_Control_XOFF          0 Rx_Control_Unknown_Opcodes    0 PHY_Media_Type      1 max_frame_size      1522 CTRL                00100241 STATUS              00280783 TXCW                00000000 RXCW                00000000 avd/ctrl            0de1/0200 Check the output for (RX/TX and other) errors.  If there are errors/drops, wait a awhile then perform the diag test again to see if the error counters increase.

If you need to set/force the speed/duplex on an interface, use:

config system interface edit "<WAN interface name>" set speed <value> next end Where speed value is auto        Automatically adjust speed. 10full      10M full-duplex. 10half      10M half-duplex. 100full     100M full-duplex. 100half     100M half-duplex. 1000full    1000M full-duplex.

[Toshi_Esumi]

If it was working fine before, I would say mostlikely a duplex mismatch or two, either or both WAN side interface to the ISP's device and LAN side interface to a switch.

[Ckny]

Is there a report I can generate to see what my historical duplex and speed settings have been on the Fortinet firewall? Is there a report to show past or current duplex mismatches?

[Toshi_Esumi]

I don't know the historical part unless it's managed by FortiManager. But duplex mismatch needs to be determine by checking 1) what's configure on both side, 2) what both sides ended up using if negotiated, then 3) if both sides are not matching in 2) it's a "mismatch".

For fortigate you can see, or not see, what's configured in the interface config. By default it's "auto/auto". If anything else hard-coded you would see like "set 1000full". To check what it ended up using is in "diag hard deviceinfo nic INTERFACE_NAME" like below:

xxx-fg1 # diag hard device nic internal1 Description     :FortiASIC NP6LITE Adapter Driver Name     :FortiASIC NP6LITE Driver Board           :60E lif id          :3 lif oid         :67 netdev oid      :67 Current_HWaddr   00:09:0f:09:fe:02 Permanent_HWaddr 70:4c:a5:bc:38:bb ========== Link Status ========== Admin           :up netdev status   :up autonego_setting:1 link_setting    :0 speed_setting   :10 duplex_setting  :0 Speed           :1000 Duplex          :Full link_status     :Up ============ Counters =========== Rx Pkts         :456459759 Rx Bytes        :398650410228 Tx Pkts         :337365528 Tx Bytes        :69211843305 Host Rx Pkts    :390926245 Host Rx Bytes   :9857764254 Host Tx Pkts    :275627811 Host Tx Bytes   :8101527118 Host Tx dropped :0

You need to check the same on the other side of the cable. In case if you can't check or suspect what the other party, like an ISP, is NOT telling the truth, you should set "auto/auto" then check what your FGT ended up using. For the speed there is a way to detect without negotiation so even if the other end is hard-coded it would match. However, duplex can't be detected without negotiation. So FGT side should end up with "half". Then you now know you have to hard-code on FGT side.

[rwpatterson]

During a maintenance window, I would suggest running through all the speed options and see what works. This will bypass what the ISP may or may not tell you. Also, if you connect at half duplex, your speed will be appreciably slower.

[Toshi_Esumi]

correction to my last post:

wrong: set 1000full

correct: set seed 1000full

[rwpatterson]

toshiesumi wrote:

correction to my last post:

wrong: set 1000full

correct: set seed 1000full

correct: set speed 1000full, unless you are sowing crops....

[Toshi_Esumi]

Thank you for your correction to my correction. I appreciate it

[Ckny]

Thank you for the great information. I will check if there is a FortiManager tracking history so we can reference. Chief firewall tech had told me new faster ISP would be solution. I switched ISP but the problem remained. Here is the pattern: we reset or reinstall device and there are no problems, all looks good. Then over 24-72 hours speeds dive and dropouts increase until unusable. This happened across devices: when the Fortinet firewall was first installed replacing the functioning existing firewall, when the VOIP router was replaced as part of troubleshooting, and when we got the new faster ISP service and modem. Same pattern. The new service went from blazing to below 56k-modem performance. Does this point to duplex mismatches or perhaps something else? I know the Firewall tech was troubleshooting with traffic shaping, for example. Where I stand now: After new ISP didn't solve, Firewall tech appeared and resolved problem quickly, even though his colleagues had been working on it for days prior with no success. Problem has not resurfaced in the weeks since. The tech will not tell me what he did to fix it (I can guess to cover error or incompetence). I want to know so it won't happen again. I will ask about FortiManager monitoring and logs. Insight?