Hot![solved] strange behaviour in ipsec site2site (was a p1 auto-negotiation issue)

Author
sw2090
Platinum Member
  • Total Posts : 551
  • Scores: 39
  • Reward points: 0
  • Joined: 2017/06/14 01:27:25
  • Location: Regensburg
  • Status: offline
2020/03/04 03:51:41 (permalink)
0

[solved] strange behaviour in ipsec site2site (was a p1 auto-negotiation issue)

Hi,
 
I have the following setup here:
 
Side A:
 FGT 100E with 6.0.9
 lan subnet is xx.xx.xx.0/24
 
Side B:
 FGT 80C with 5.6.12 (latest available for 80C)
 lan subnet is yy.yy.yy.0/24
 
IPSec Site2Site between A and B is set up.
 
The Tunnel is up (green in monitor) and monitor does show there is traffic in both directions on this one.
 
now when I ping xx.xx.xx.10 from client yy.yy.yy.50 on Side B I get no response.
Flow Trace on Side B shows that the traffic goes out to side A over the IPsec as it should.
Flow Trace on Side A shows that the traffic coming from my client does reach side A and is routed on correctly
Flow Trace on Side A also shows me that there is traffic from xx.xx.xx.10 to my client and that it is routed over the Ipsec correctly.
 
However: Flow Trace on Side B with Filter saddr xx.xx.xx.10 and daddr yy.yy.yy.50 shows just nothing.
So looks to me as if the traffic coming back to my client on side B gets lost somewhere on the ipsec?
 
Did anyone here have this? Any clues?
 
Thnx in advance!
Sebastian
post edited by sw2090 - 2020/03/06 01:50:24
#1

12 Replies Related Threads

    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/04 06:55:55 (permalink)
    0
    hm it looks like as if i solved it myself.
     
    first I found that NAT-T was not enabled on one side but enabled on the opposite side. 
    So I enabled it and brought the tunnel down on this side to have it re-establish itself automagically which it did.
    The behaviour was still the same...
     
    Then I found some hint on Mike's Fortinet Guru Page. The was a paragraph about restarting ike and clearing old gateways.
    So I did 
     
    diag vpn ike restart
    diag vpn ike gateway clear
    and after those:
    diag vpn ike status
    this showed me that one ipsec tunnel was up again. There is two others but those have different policies and routes etc so they don't affect this one. Those are not up yet because I still have to create their remote end ;)
     
    after this I tried pinging again and...magically...it works now....
    #2
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/04 07:08:27 (permalink)
    0
    hm it works from B to A now. Even remote desktop works from Client on Side B to Server on Side A.
    But if I try to ping from A to B my traffic gets dropped (denied by forward policy check (Policy #0) even though there devinitely is a matching policy in place (it showed me it matched it in flow debug before). 
    It finds the correct route but seems not to match my policy anymore...
     
    #3
    emnoc
    Expert Member
    • Total Posts : 5546
    • Scores: 357
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/04 07:26:33 (permalink)
    0
    You need to double check policy and possible  "diag sniffer packet <tunnel_name>"
     
    BTW NAT-T is negotiated at phase1 and HAS NOTHING to do with phase2 which are SA ....i.e layer3 proxy-ids
     
    I would double check policy , routes and ensure NAT is off if not required.
     
    Ken Felix
     

    PCNSE 
    NSE 
    StrongSwan  
    #4
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/04 07:27:17 (permalink)
    0
    correction again :)
     
    That it worked in one direction only was due to the client on side A which i used to ping side B.
    This client is in two subnets and so has two default routes and their metric struck me this time ;)
    On that client that is wanted behaviour ;)
     
    I tried on one that is only in xx.xx.xx.0/24 and it worked from there as it should ;)
     
    I consider this last one to be a layer-8-fail :D
     
    #5
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/04 07:31:46 (permalink)
    0
    never said NAT-T had to do something with phase 2. p2 was up fine all the time accoarding to the output of diag vpn ike status...
    I just noticed it wasn't turned on on Side B but should be since this is behind a LTE Router so I turned it on...
     
    #6
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/05 05:14:35 (permalink)
    0
    Right,
     
    new problem now:
     
    ipsec is up and working fine so far.
    Now if I reboot remote end (side B) tunnel goes down (of course)..ok so far :)
    remote end comes up again but tunnel is still down and takes ages until it comes up again.
    Once it is back up again it works fine again.
    When it goes down the FGT on side A shows it is down in ipsec monitor on gui.
    When side B comes up again I see p1 connection requests from it on side A FGT in diag debug app ike -1 output.
    It does match p1 proposals and the right tunnel but then don't go further.
    To mee it looks as if when the ipsec goes down when side B reboots it does get this (I assume either via DPD oder NAT Keepalive - its shown down in ipsec monitor) something is leftover and that prevents side B from establishing a new vpn conncetion until it gets removed by timeout.
    Maybe it does establish P1 but then tries to re-establish old p2 which will not work until some timeout kicks it?
     
    Does anyone have any tip or idea for me?
    #7
    emnoc
    Expert Member
    • Total Posts : 5546
    • Scores: 357
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/05 08:48:51 (permalink)
    0
    Enable auto-negotiation on the phase2 definitions and the ipsec tunnel will always come up. You do it from the cli
     
       https://kb.fortinet.com/kb/documentLink.do?externalID=12069
     
    The diag vpn tunnel list will show you want ones are not enable in the output.
     
    Ken Felix

    PCNSE 
    NSE 
    StrongSwan  
    #8
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/05 09:35:17 (permalink)
    0
    Ken,
     
    auto negation is always on on all my ipsec tunnels here on both side. I see in the logs that both sides try to auto negotiate. Also once that Tunnel finally has come up and I bring it down with the button in ipsec manager on gui it does go down and re-auto-negotiates within some sec.
    It just does not do this when I reboot side B.
     
    #9
    emnoc
    Expert Member
    • Total Posts : 5546
    • Scores: 357
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/05 12:57:25 (permalink)
    0
    How did you determine that in the logs? If one side goes down, does the other side  clears the ike and ipsec SAs? 
     
      diag vpn tunnel list | grep spi
      diag vpn ike gateway
     
    If you still see IKE/IPSEC SAs than one side is stuck.  I would suggest  diagnose DPD  if you suspect DPD
     
    diag debug reset 
    diag debug enable 
    diag debug application  ike -1 
    diag debug enable
     
     
    Monitor R U THERE messages
     
    Alternative, run diag sniffer packet on wan interface and just for the interval and packets
     
    e.g
     
        diag sniffer packet wan "host 192.0.2.1 and port 500 or 4500"
     
     
    Ken Felix
     

    PCNSE 
    NSE 
    StrongSwan  
    #10
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/05 23:49:10 (permalink)
    0
    hm diag debug gateway list lists two gateways on both sides (one for each direction on each side).
    diag debug app ike -1 shows there is IKE SA Requests on both sides that are answered since it is matching proposals and gatway correctly. It also right afterwarts does NAT-T and NAT-D handshake successfull.
    Right after this nothing more happens until either one SIDE retransmits or receives a RETRANSMIT query from the opposite side.
    thus it stays stuck until after like 15mins.
    DPD seems to work since I can see the tunnel goes down like 15sec after I entered exec reboot on the other side which seem sto meet the default DPD timeouts and retry count.
     
    Maybe auto negation is the belzebub here too, since side A tries to re-autonegotiate the tunnel as soon as has gone down but opposite side is not yet there again. Could that cause dead SA's under some weird circumstances?
     
    I will keep on searching...
    #11
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/06 00:35:55 (permalink)
    0
    this is what diag vpn tunnel list reports on side A:
     
    name=d_e1-messe_e1 ver=2 serial=37e xx.xx.xx.xx:4500-><remotegwip>:28919
    bound_if=7 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/0
    proxyid_num=1 child_num=0 refcnt=10 ilast=1 olast=1 ad=/0
    stat: rxp=0 txp=0 rxb=0 txb=0
    dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=15413
    natt: mode=keepalive draft=0 interval=10 remote_port=28919
    proxyid=t_d_e1-messe_e1 proto=0 sa=1 ref=2 serial=2 auto-negotiate
     src: 0:0.0.0.0/0.0.0.0:0
     dst: 0:0.0.0.0/0.0.0.0:0
     SA: ref=3 options=18227 type=00 soft=0 mtu=1438 expire=7414/0B replaywin=1024
     seqno=1 esn=0 replaywin_lastseq=00000000 itn=0
     life: type=01 bytes=0/0 timeout=14128/14400
     dec: spi=f1e85cd2 esp=3des key=24 73a5480aead7101386c92d4ebbb97da4eefbc1b320d80038
     ah=sha1 key=20 f5621e3f01ae26136ee0f78d74ff3f9630f002d8
     enc: spi=a7634197 esp=3des key=24 cb82ef81d9b0cc41f5dda109594966bc94882a84bf65a4c9
     ah=sha1 key=20 2b4793be6cfca7e1d6f1be77f54c66f00d8997d6
     dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
     npu_flag=00 npu_rgwy=<ipofremotegw> npu_lgwy=xx.xx.xx.xx npu_selid=342 dec_npuid=0 enc_npuid=0

     
    On Side B it looks equally (except from tunnel name and gw ips of course).
    This is when the tunnel is up and running btw.
     
    I see one more difference btw: Side A has dpd timeout 5000ms and retry count 3 while Side B has timeout 20000ms and retry count 3.
     
    Done some more Debugging:
     
    I can see in ike debug log on side a that dpd detects a link 5-10sec after I triggert a FGT reboot on side B. It shuts the tunnel down, deletes sa and flushes the tunnel. It also sends a snmp trap.
     
    So DPD seems to work properly here.
    post edited by sw2090 - 2020/03/06 01:20:29
    #12
    sw2090
    Platinum Member
    • Total Posts : 551
    • Scores: 39
    • Reward points: 0
    • Joined: 2017/06/14 01:27:25
    • Location: Regensburg
    • Status: offline
    Re: strange behaviour in ipsec site2site 2020/03/06 01:49:38 (permalink)
    0
    ok I seem to have found the corpus delicti now.
    It is indeed caused by auto-negotiotiation. But not by auto-negotiation in p2, indeed it happens in p1.
    And it is caused on Side A.
     
    In fact DPD flushes the tunnel when it detects the link fail when I reboot FGT on side B.
    Right after that p1 auto-negotiate tries to re-establish the tunnel which will fail because side B is not yet up again. This seems to create a dead connection on side A. Oncoming retries keep using this dead connection until it timeout.
     
    I now disabled p1 auto-negotiation on Side A (I don't need side A to auto negotiate anyways). So Side B has to establlish the tunnel once it is back up again which is triggered by Side B p1 auto-negotiation. Since there is no connections for this tunnels on Side A yet then the tunnel fires up asap and works fine.
     
    Up to now I didn't even know there is a auto-neg feature in p1. Articles I found keep referring to p2 only and gui once more don't show it at all :/
     
    Alas thanx guys this is solved so far :)
    #13
    Jump to:
    © 2020 APG vNext Commercial Version 5.5