LACP recommandation between Fortigate FortiOS 5 and Cisco switch

Author
miky
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2011/10/23 14:57:23
  • Status: offline
2014/02/20 14:42:24 (permalink)
0

LACP recommandation between Fortigate FortiOS 5 and Cisco switch

Hello,

I would like to know if some of you have a recommendation for a configuration between a Cisco switch port-channel and a Fortigate Agg FortiOS5

On my Cisco configuration I' ve used this for the physical interfaces
channel-group 1 mode active
switchport nonegotiate

On the Fortigate I have
edit " Agg1"
set vdom " root"
set type aggregate
set member " port1" " port2"
set lacp-mode passive

So LACP active on the Cisco switch and passive on the Fortigate.

Thank you
#1

19 Replies Related Threads

    Matthew Mollenhauer
    Silver Member
    • Total Posts : 69
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/07 20:06:48
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/20 15:04:51 (permalink)
    0
    The below are the configs we' re using:

    Cisco:
    interface Port-channel1
    description uplink to FortigateFW
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 100-150,200-250,300-350
    switchport mode trunk
    spanning-tree portfast trunk
    end


    Fortigate:
    config system interface
    edit " LACP VLAN Group"
    set vdom " Blah"
    set type aggregate
    set member " port28" " port29"
    set snmp-index 52
    set lacp-mode static
    next
    end


    The Cisco switches we' re connecting with are stacked 3750G' s running IOS 15.0(2)SE

    Our Fortigates are a HA Pair (A/P) 1240B' s running 5.0.6, though we' ve used this config since FortiOS 4 MR2.

    If you' re using HA you' ll need separate Port Channel groups for each Fortigate.

    We' ve also had one or two occurrences where we' ve had Speed/Duplex mismatches, so you may need to statically set ports on both sides.

    Regards,
    Matthew
    #2
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/20 22:06:01 (permalink)
    0

    Cisco:
    interface Port-channel1
    description uplink to FortigateFW
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 100-150,200-250,300-350
    switchport mode trunk
    spanning-tree portfast trunk
    end


    Fortigate:
    config system interface
    edit " LACP VLAN Group"
    set vdom " Blah"
    set type aggregate
    set member " port28" " port29"
    set snmp-index 52
    set lacp-mode static
    next
    end



    The cisco stuff you posted is NOT a lacp bundle btw.

    Here' s a real LACP mode active from a 3750G


    int range gi 1/0/1-2
    no shut
    switchport
    channel-group 10 mode active
    channel-protocol lacp
    load-interval 30
    logging event link-status
    logging event bundle-status
    !
    !
    int port 10
    description 2 GIG bundle to FGT
    !



    Keep in mind you can trunk over the etherchannel also. So this will allow you to use the aggregate ports more effective and by issuing sub-intf


    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #3
    miky
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/10/23 14:57:23
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 04:52:11 (permalink)
    0
    Hello,

    Thank you both for your answers.
    Regarding my Cisco configuration I just wrote the important lines (not the syslog or load-interval related commands) and of course I do use trunk with the firewall.

    I didn' t write the channel-protocol lacp command but according to the 2960X Cisco switch it is LACP.
    Group Port-channel Protocol Ports
    ------+-------------+-----------+-----------------------------------------------
    1 Po1(SU) LACP Gi1/0/41(P) Gi2/0/41(P)

    I' m asking the question because I see a lot of output drops on the port-channel interface and I would tend to think it' s due to the Fortigate side.
    #sh int po1 | i drops
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 99837
    Incremendation happen when there is a load on the link


    #4
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 06:06:10 (permalink)
    0
    highly doubt it' s the fortigate causing output drop on a interface facing the fortigate and on a cisco swith.

    Can you confirm flow-control is disable on interface gi 1/0/41/+2/0/41?

    Do you have any QoS enabled ( policy-map, thresholds,etc....)

    Do you have giant frames allowed?

    Drops could be anything from a ACL list , unknown datagrams,etc...?



    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #5
    ShrewLWD
    Gold Member
    • Total Posts : 147
    • Scores: 3
    • Reward points: 0
    • Joined: 2009/04/23 08:16:54
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 06:26:45 (permalink)
    0
    hi everyone,

    this past weekend we had issues with attempting to LAG link the SFP ports of a 100D (v506) to use dual fiber down to Cisco 3750s at our DR site.

    No matter what we tried, LAG would not link up. We are using the simplest of LAG setups on both ends.

    When I ran
    diagnose netlink aggregate name [MyLAGsName]

    I noticed
    LACP state: negotiating
    actor state: ASAIDD
    partner state: ASAOEE

    It seems Active/Active doesn' t work, and the Fortinet won' t respond either, if it is in passive mode and the Ciscos in Active. We were able to get it up and running by putting both into STATIC mode. I realize it is not the most ideal, but packets are passing, and we are not seeing any errors or dropped packets. The Fortinet TAC mentioned a known bug with some portion of LACP that is still unresolved (bug# 0229638).

    Hope this helps!
    #6
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 07:22:41 (permalink)
    0
    That' s interesting, all of the fortinet that I' ve worked with that support 802.3ad works regardless of if they are ACT/PASS

    Remember with within any 802.3ad setup, one member-side must be in ACTIVE mode. Are you 100% sure the 3750 was setup for LACP and not PAGP ? and did they issues any show etherchannel and show lacp commands ?

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #7
    ShrewLWD
    Gold Member
    • Total Posts : 147
    • Scores: 3
    • Reward points: 0
    • Joined: 2009/04/23 08:16:54
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 07:35:54 (permalink)
    0
    Hi emnoc,

    Here is what the Cisco looked like;
    !
    interface Port-channel25
    switchport access vlan 10
    !
    interface GigabitEthernet1/1/1
    description " OWS Links"
    switchport access vlan 10
    mls qos trust dscp
    channel-protocol lacp
    channel-group 25 mode active
    !
    interface GigabitEthernet2/1/1
    description " OWS Links"
    switchport access vlan 10
    mls qos trust dscp
    channel-protocol lacp
    channel-group 25 mode active
    !


    Here was my Fortinet;

    config vdom
    edit VD_PWAN
    config system interface
    edit " port15"
    set vdom " VD_PWAN"
    set type physical
    set snmp-index 25
    next
    edit " port16"
    set vdom " VD_PWAN"
    set type physical
    set snmp-index 26
    next
    edit " COLO_Link"
    set vdom " VD_PWAN"
    set broadcast-forward enable
    set l2forward enable
    set stpforward enable
    set type aggregate
    set member " port15" " port16"
    set snmp-index 44
    next
    end

    The only settings that allowed it to link up was;
    Cisco:
    channel-group 25 mode on
    Fortinet:
    set lacp-mode static

    We tried all the other variations, with a full reboot of the 100D between changes.
    #8
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 07:42:40 (permalink)
    0
    You really need the show lacp commands.

    What you have with the " lacp-mode-static" is a static bundling no LACP protocol or no-Negotiation. No-Negotiation is not necessary a bad thing, and some device just don' t do LACP that good. I never heard of a cisco or fortigate being one of those devices tho.

    I' m curious, did you do this without the cisco config of QoS enabled and without the fortigate configuration of;


    set broadcast-forward enable
    set l2forward enable
    set stpforward enable




    I' m assuming this is a transparent L2 firewall ? I never done lag bundles within any Layer2 transparent-firewalls but I don' t see why it would make a difference.


    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #9
    ShrewLWD
    Gold Member
    • Total Posts : 147
    • Scores: 3
    • Reward points: 0
    • Joined: 2009/04/23 08:16:54
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 08:27:02 (permalink)
    0
    I must confess to not being proficient in Cisco IOS anymore, so we had Cisco TAC SSH' d into the 3750s while I made changes to the Fortinet on my end. He was watching running lacp commands, including show lacp event, show lacp 25 (counters and internal detail), show spanning-tree, etc.

    We did not try turning off QoS, nor turning off the Fortinet ' sets' . We don' t do non-IP traffic, so we don' t need the l2 line, and I don' t believe we need the stpforward either, since there are no additional cisco devices behind the Fortinet needing to send traffic down. I was just turning those on thinking they were causing the lacp to fail originally, before getting Fortinet and Cisco TACs involved.

    Yes, the VD_PWAN is a transparent VDOM, with VDOMLinks from several other transparent VLANs accessing it. With only port15 enabled, and all firewall rules pointed to it, traffic passed no problem. Same with switching to 16. As soon as those two were LAGd (and firewall policies updated to point to the LAG), all traffic stopped. Additionally, when we put it back the way it was, for a period of time (5-10 minutes) traffic still did not flow.
    #10
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 08:51:25 (permalink)
    0
    It sounds like a L2 STP issue if I had to guess. Once again I never seen a transparent firewall using LACP bundle.

    I think fortinet TAC could clarify if this is acceptable with the config you have and with the with the cmd;

    set l2forward enable
    set stpforward enable

    Both of which should be used with caution. I ' m betting most of your problems are in these 2.

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #11
    ShrewLWD
    Gold Member
    • Total Posts : 147
    • Scores: 3
    • Reward points: 0
    • Joined: 2009/04/23 08:16:54
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 10:35:32 (permalink)
    0
    Possibly, and, because I have a known good configuration from each end, I will certainly take a stab at removing those lines and putting both devices back into ACTIVE - ACTIVE, but before I got the TACs involved, those two lines weren' t in there. So unless the Cisco TAC found an additional setting he changed (possible), I don' t see it making a difference.
    The configs changed in this order (none worked except the last);
    both devices were set to ACTIVE ACTIVE
    I added those lines
    I tried Active Passive
    I tried Passive Active
    I had read on a recent previous lacp discussion that STATIC is not a good idea, so I didn' t try that, I got the TACs involved instead.
    We tried static static

    So those lines were added early in my tests, and again, I don' t know if the Cisco TAC found and changed something else. But ' I will submit myself to these experiments in the name of science!' -Allan Walker Blair ;-)
    #12
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 11:05:55 (permalink)
    0
    FWIW:

    You should open a new thread and not hijack this one. But I would review your configuration to ensure the LACP neighbors counters

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #13
    ShrewLWD
    Gold Member
    • Total Posts : 147
    • Scores: 3
    • Reward points: 0
    • Joined: 2009/04/23 08:16:54
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/25 11:49:01 (permalink)
    0
    How is this a hijack?
    1)The OP requested recommendations for Cisco to Fortinet LAG. That' s what I have, and this have been my experience, and what works.
    2) I' m even willing to continue troubleshooting for the sake of the community in general, and the OP specifically, so that he can have an even more informed decision.
    #14
    miky
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/10/23 14:57:23
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/26 06:03:09 (permalink)
    0
    Hello,

    I don' t mind people talking about the LACP counters problem on this thread because I asked the Cisco vs Fortigate configuration recomendation as I see output drops on the Cisco side. The latter statement is my real problem.

    emnoc: My configuration is vers simple, I have no QoS enabled except the command mls qos globally and mls qos trust cost at the interface level. I didn' t allow giant frames but usually giant frames show up in the " giant" counter
    #sh int po1 | i giant
    0 runts, 0 giants, 0 throttles

    And there is no ACL. Unknown datagrams might show up in the " unknown protocol drops" counter
    #sh int po | i unknown protocol drops
    0 unknown protocol drops

    My assumption towards the Fortigate is biaised I agree. That' s because I have been working for a decade with Cisco switches and much much less with Fortigates. And the FortiOS 5 is buggy, it keeps crashing !!! (case opened)

    So I confirm that Cisco is using LACP protocol
    #sh etherchannel 1 summary
    Number of channel-groups in use: 2
    Number of aggregators: 2
    Group Port-channel Protocol Ports
    ------+-------------+-----------+-----------------------------------------------
    1 Po1(SU) LACP Gi1/0/46(P) Gi2/0/46(P)

    On the Fortigate
    # diagnose netlink aggregate name Agg1
    LACP mode: passive
    LACP speed: slow
    LACP HA: enable

    slave: port1
    actor state: PSAIEE
    partner state: ASAIEE
    aggregator ID: 2

    slave: port2
    actor state: PSAIEE
    partner state: ASAIEE
    aggregator ID: 2


    #show lacp 1 counters
    LACPDUs Marker Marker Response LACPDUs
    Port Sent Recv Sent Recv Sent Recv Pkts Err
    ---------------------------------------------------------------------
    Channel group: 1
    Gi1/0/46 54870 50720 0 0 0 0 0
    Gi2/0/46 54868 50723 0 0 0 0 0


    My firewall is not transparent, it' s a L3 firewall so it should block bpdu' s

    My next test will probably be to configure the port-channel static without LACP. I' ll keep you updated.
    #15
    emnoc
    Expert Member
    • Total Posts : 4442
    • Scores: 255
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/26 06:57:26 (permalink)
    0
    For the output drops are going to be cisco specific. I would start by ensuring flow control is not enable



    show int por 1 flow


    These 2 commands might help with showing you the number of process switched packets.



    show interface stats
    show interfaces switching



    e.g



    SW1#show interface port 1 switching
    Port-channel1 Etherbundle to MACORE1
    Throttle count 0
    Drops RP 0 SP 0
    SPD Flushes Fast 0 SSE 0
    SPD Aggress Fast 0
    SPD Priority Inputs 0 Drops 0

    Protocol Path Pkts In Chars In Pkts Out Chars Out
    Spanning Tree Process 0 0 23025 2210400
    Cache misses 0
    Fast 0 0 0 0
    Auton/SSE 0 0 0 0
    SW1#



    I would disable anything that' s process switch ( cdp, lldp, dynamic routing protocols, GRE tunnels, basically anything sourced from the switch directly)



    And Yes I realize this is not a transparent. The other gentlemen is using LACP in a layer2 firewall configuration.


    Now for some more Qs:


    Do you see any performances with users traffic or just monitoring the output drops?

    Have you tried to disable any QoS as temporal fix to see if that effects anything or decreases the drops?

    Do the drops increase/decrease if you check all of the above items mention above ?


    FWIW, the drops probably has nothing to do with LACP btw and I disagree that 5.0.x is buggy. If it is than we are not effected by anything

    if it was as buggy as some claims than none of the 18 devices I mention that' s on all versions of 5.0.x would be buggy.

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #16
    miky
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/10/23 14:57:23
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/26 09:04:29 (permalink)
    0
    emnoc, thank you for your answer.

    I had forgotten to mention that flow control is disable by default (input direction) on Cisco not available on the output direction.
    http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960/software/release/12-2_58_se/configuration/guide/2960scg/swint.html#wp2071785

    #sh int Gi1/0/46 flowcontrol
    Port Send FlowControl Receive FlowControl RxPause TxPause
    admin oper admin oper
    --------- -------- -------- -------- -------- ------- -------
    Gi1/0/46 Unsupp. Unsupp. off off 0 0
    Gi2/0/46 Unsupp. Unsupp. off off 0 0

    LLDP is not enabled, CDP is disabled on the physical interfaces connected to the Fortigate, there are no routing protocol and no gre.
    The switch is pure L2 and the Fortigate is taking care of the L3

    #show interfaces port-channel 1 switching
    Port-channel1 FG_Agg1

    Protocol Spanning Tree
    Switching path Pkts In Chars In Pkts Out Chars Out
    Process 0 0 19405256 1238882128
    Cache misses 0 - - -
    Fast 0 0 0 0
    Auton/SSE 0 0 0 0

    NOTE: all counts are cumulative and reset only after a reload.

    #show interfaces port-channel 1 stats
    Port-channel1
    Switching path Pkts In Chars In Pkts Out Chars Out
    Processor 0 0 7260046 492528964
    Route cache 0 0 0 0
    Total 0 0 7260046 492528964

    Do you see any performances with users traffic or just monitoring the output drops?
    => Yes users complain about slow network (as usual you' ll tell me) but that how I ended searching on the root cause of these output drops

    Have you tried to disable any QoS as temporal fix to see if that effects anything or decreases the drops?
    => No but I will add it in my action plan. However, I just wanted the switch to keep the marking. And also I see no reason why 100M traffic would be queued on a 2x1G port-channel


    About the 5.0, trust me =) There is at least one real nasty bug on it, maybe related with dhcp relay.
    In many cases we had kernel panic, we are able to reproduce it though and we' re currently investigating this.
    #17
    Matthew Mollenhauer
    Silver Member
    • Total Posts : 69
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/07 20:06:48
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/02/27 16:22:52 (permalink)
    0

    The cisco stuff you posted is NOT a lacp bundle btw.


    You' re absolutely correct, it' s not LACP but raw/static etherchannel.

    We had several issues when we did our deployment where the Cisco and Fortigate would either not negotiate at all or it would negotiate too often and drop the link. Changing to use a static link aggregation was the best solution in our case, though it' s not the only way aggregation can be done. That' s also the reason the interface is labeled " LACP VLAN Group" it was originally a proper LACP configuration.

    I know some people argue against using static aggregation because there are some dangers with MAC flapping & loops, but in a DC environment where physical connections are static(we' ve made no physical changes to our 1240B' s in 3.5 years) the dangers are minimal. IMO, LACP introduces a bigger risk where a software bug can cause the negotiation to not work properly, ie see ShrewLWD post that mentions bug #0229638.


    As for the stability of 5.0, I' d have to agree that there have been several bugs that could have been nasty to our Production environment. 5.0 GA -> 5.0.2 were not " friendly" while 5.0.3 was actually somewhat mature. I guess that' s why we send firmware changes through our Test and DR environments before they hit Production. I also have to say I' m not overly happy with the way that new features and changes of functionality are introduced in the minor releases, they should be bug fixes only.

    Regards,
    Matthew Mollenhauer
    #18
    miky
    New Member
    • Total Posts : 7
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/10/23 14:57:23
    • Status: offline
    RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2014/03/05 05:38:44 (permalink)
    0
    Hello,
    Just to let you know, the port-channel is now static/mode on but we still have output errors seen on the Cisco Po
    Po1 => 0,022% (expressed in packets)
    Gi1/0/46 => 0,017 %
    Gi2/0/46 => 0,039 %
    I know the percentage of error is quite low but there just should not have these errors in theory.
    Regards
    #19
    Ed
    New Member
    • Total Posts : 5
    • Scores: 0
    • Reward points: 0
    • Joined: 2016/08/19 11:10:08
    • Status: offline
    Re: RE: LACP recommandation between Fortigate FortiOS 5 and Cisco switch 2016/08/20 03:27:38 (permalink)
    0
    Do you still need separate ether-channels on the Cisco side if the cluster is Active Active?
    #20
    Jump to:
    © 2018 APG vNext Commercial Version 5.5