Fortigate dropping internal network and external network packets.

Author
Rait
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/02/20 00:34:31
  • Status: offline
2015/03/12 23:44:55 (permalink)
0

Fortigate dropping internal network and external network packets.

Hello!
I have a issue with fortigate 800c. It keeps dropping packets on high bandwidth load. Lets say I ping a 10.2.2.X server from 10.2.2.X server with 54 bytes, I lose around 5 packets in 30 seconds. When I increase the packet size to 65000 it loses 10-15 packets in 15 seconds.
I have debugged from console and nothing is blocked or dropped. Sniffer shows all packets are "ok", but I guess firewall drops them before it can process them. Switches are ok, cables are ok etc.
 
This is a new unit in our production so I tried to switch back to old Juniper and that did not drop packets at all. So it has to be the fortigate unit.
 
MTU on ports is default (1500 I guess).
 
Where could the problem be?
#1

4 Replies Related Threads

    emnoc
    Expert Member
    • Total Posts : 5208
    • Scores: 339
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Fortigate dropping internal network and external network packets. 2015/03/13 02:44:43 (permalink)
    0
    I would start by looking at port errors. Sounds like duplex mismatch I like a few important things to look at in the below output but any errors or collision, is a good indicator of duplex mis-matches from the diag hardware dev nic < nicname >
     

    ADFGT110C # diag hardware deviceinfo nic wan1
    Description            Broadcom 570x Tigon3 Ethernet Adapter
    Part_Number            BCM95786T8600
    PHY_Device_ID            5787
    Driver_Name            tg3
    Driver_Version            3.85l
    PCI_Vendor            0x14e4
    PCI_Device_ID            0x169a
    PCI_Subsystem_Vendor        0x14e4
    PCI_Subsystem_ID        0x969a
    PCI_Revision_ID            0xb002
    PCI_Address            2:0.0
    Memory                0xdfd00000
    IRQ                10
    System_Device_Name        wan1
    Current_HWaddr            00:09:0f:09:01:08
    Permanent_HWaddr        00:09:0f:ce:42:0f
    Link                up
    Speed                1000 Mbps
    Duplex                full
    FlowControl            Tx off, Rx off
    MTU_Size            1500

    Rx_Packets            1143337352
    Rx_Packets_Dropped        0
    Tx_Packets            1021102004
    Rx_Bytes            548759472
    Tx_Bytes            3298808021
    Rx_Errors            0
    Tx_errors            0
    Multicast            1601792
    Collisions            0
    Rx_Length_Errors        0
    Rx_Over_Errors            13
    Rx_Frame_Errors            0
    Tx_aborted_Errors        0
    Tx_carrier_errors        0
    Rx_CRC_Errors            0
    rx_pending            200
    tx_pending            511
    tg3_flags            8248ec05
    tg3_flags2            380c9200
    tg3_flags3            00002000
    rx_rcb_ptr            00000054
    rx_producer            00000054


     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #2
    Rait
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/02/20 00:34:31
    • Status: offline
    Re: Fortigate dropping internal network and external network packets. 2015/03/13 04:31:49 (permalink)
    0
    emnoc
    I would start by looking at port errors. Sounds like duplex mismatch I like a few important things to look at in the below output but any errors or collision, is a good indicator of duplex mis-matches from the diag hardware dev nic < nicname >
     

    ADFGT110C # diag hardware deviceinfo nic wan1
    Description            Broadcom 570x Tigon3 Ethernet Adapter
    Part_Number            BCM95786T8600
    PHY_Device_ID            5787
    Driver_Name            tg3
    Driver_Version            3.85l
    PCI_Vendor            0x14e4
    PCI_Device_ID            0x169a
    PCI_Subsystem_Vendor        0x14e4
    PCI_Subsystem_ID        0x969a
    PCI_Revision_ID            0xb002
    PCI_Address            2:0.0
    Memory                0xdfd00000
    IRQ                10
    System_Device_Name        wan1
    Current_HWaddr            00:09:0f:09:01:08
    Permanent_HWaddr        00:09:0f:ce:42:0f
    Link                up
    Speed                1000 Mbps
    Duplex                full
    FlowControl            Tx off, Rx off
    MTU_Size            1500

    Rx_Packets            1143337352
    Rx_Packets_Dropped        0
    Tx_Packets            1021102004
    Rx_Bytes            548759472
    Tx_Bytes            3298808021
    Rx_Errors            0
    Tx_errors            0
    Multicast            1601792
    Collisions            0
    Rx_Length_Errors        0
    Rx_Over_Errors            13
    Rx_Frame_Errors            0
    Tx_aborted_Errors        0
    Tx_carrier_errors        0
    Rx_CRC_Errors            0
    rx_pending            200
    tx_pending            511
    tg3_flags            8248ec05
    tg3_flags2            380c9200
    tg3_flags3            00002000
    rx_rcb_ptr            00000054
    rx_producer            00000054


     


    Output on port4 that has vlan interfaces.
    fw1-tix # diagnose hardware deviceinfo nic port4
    Description     :FortiASIC NP4 Adapter
    Driver Name     :FortiASIC Unified NPU Driver
    Version         :1.0
    PCI Slot        :01:00.0
    PCI_Revision_ID :1
    Board           :fgt800c
    SN              :FG800C3914801794
    Major ID        :19
    Minor ID        :0
    LIF ID          :5
    NPU OID         :5
    NPU OID_VID     :7
    netdev flags    :0x00001303
    Current_HWaddr   00:09:0f:09:00:14
    Permanent_HWaddr 08:5b:0e:96:7a:8d
    Queue           :On
    rx_buffer_len   :2048
    max_frame_size  :1522
    min_frame_size  :278
    MTU             :1500
    Hidden          :No
    dyn_lif         :0
    npu_cap         :0000600b
    ses_mask        :40027dcb
    cmd_in_queue    :0
    half_id         :1
    phy_addr        :0x07
    phy_flags       :0x4002
    medium_type     :Copper
    sw_port         :9
    sw_np_port      :29
    sw_stats_port   :0
    phy_oid         :0x05
    slot_id         :0x00
    vid_phy[6]      :[0007][0000][0000][0000][0000][0000]
    vid_fwd[6]      :[0000][0000][0000][0000][0000][0000]
    oid_fwd[6]      :[0000][0000][0000][0000][0000][0000]
    ========== Link Status ==========
    Admin           :up
    PHY Status      :up
    PHY Speed       :1000
    Duplex          :Full
    link_status     :1
    rx_link_status  :1
    remote_fault    :0
    local_fault     :0
    local_warning   :0
    int_phy_link    :0
    int_phy_reinit_cnt:0
    link_andone     :0
    serdes_mode     :SerDes
    ============ Counters ===========
    clst_tx_orig    :0
    clst_tx_o_free  :0
    clst_tx_redir   :0
    clst_tx_rd_free :0
    clst_tx_reply   :0
    clst_tx_rp_free :0
    dd_wo_eop       :0
    tcp_udp_csum_err:49519
    rx_error        :0
    rx_crc_error    :0
    rx_len_error    :0
    rx_carrier      :0
    rx_oversize     :0
    rx_undersize    :0
    tx_collision    :0
    ip_sum_offload  :0
    ipsec_dec       :0
    ipsec_dec_drop  :0
    ipsec_antireplay_f:0
    ipsec_antireplay_p:0
    aps_log         :0
    Rx Pkts         :15840863264
    Rx Bytes        :12734562129748
    Tx Pkts         :18424647921
    Tx Bytes        :14544615864915
    Host Rx Pkts    :9289600739
    Host Rx Bytes   :10985710481858
    Host Tx Pkts    :6042606135
    Host Tx Bytes   :997450169787
    sw_rx_pkts      :2955977526
    sw_rx_bytes     :4291837538
    sw_tx_pkts      :1244797450
    sw_tx_bytes     :1870594196
    sw_rx_mc_pkts   :6801849
    sw_rx_bc_pkts   :1825246
    sw_np_rx_pkts   :18428762305
    sw_np_rx_bytes  :14618787336616
    sw_np_tx_pkts   :15844958527
    sw_np_tx_bytes  :12798395296565
    sw_np_rx_mc_pkts:15
    sw_np_rx_bc_pkts:4509951


    What would this mean:
    tcp_udp_csum_err:49519
    #3
    ashukla_FTNT
    Silver Member
    • Total Posts : 87
    • Scores: 7
    • Reward points: 0
    • Joined: 2015/02/02 04:25:16
    • Status: offline
    Re: Fortigate dropping internal network and external network packets. 2015/03/13 11:24:37 (permalink)
    0
    Rait
     
     
    What would this mean:
    tcp_udp_csum_err:49519




    The means theh checksum was wrong and the Np4 hardware acclearator (asic) dropped these many packet. As these numbers are historical number, you have to start the traffic and monitor if this number is increasing, then it is clear that NP4 is dropping because checksum is wrong.
     
    In pc (server) most of the network cards supports offloading the tcp/udp checksum calculation to the nic card. Disable this feature and try. At the same time do the capture on source pc and check the wireshark reports checksum as wrong.
     
    Please note when checksum offload is enabled to NIC card, wireshark may report checksum is bad. In reality it is because of checksum offload to NIC, the actual checksum is not wrong.
     
     
    #4
    emnoc
    Expert Member
    • Total Posts : 5208
    • Scores: 339
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Fortigate dropping internal network and external network packets. 2015/03/14 01:13:55 (permalink)
    0
    Op, if you want to get creative you can  get interface by interface stats using fsnsyctl
     
    e.g
     
     fnsysctl cat /proc/net/dev
     
    The output will be a column of
     
     Interface|       bytes    packets errs drop fifo other compressed mcast colls

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #5
    Jump to:
    © 2019 APG vNext Commercial Version 5.5