Hot!Specific traffic appears to enter IPsec tunnel but not exit!?

Author
journeyman
Gold Member
  • Total Posts : 169
  • Scores: 0
  • Reward points: 0
  • Joined: 2011/03/15 22:56:36
  • Status: offline
2018/11/15 21:20:27 (permalink) 5.4
0

Specific traffic appears to enter IPsec tunnel but not exit!?

On an ipsec interface tunnel between FGT-A and FGT-B, I see specific traffic (Server SYN) enter the ipsec interface at A but cannot see it exit at B (using diagnose sniffer packet and diagnose debug flow at each end).
The configuration and functionality is not new and is duplicated widely across our system; I see the problem only on one instance.
- The ipsec tunnel also carries OSPF and all necessary routes are visible;
- The ipsec tunnel carries all traffic to and from FGT-B;
- The Client also establishes a session in the reverse direction, this is successful.
- I see no other functionality impacted on the tunnel
- I have checked firewall address and policy configurations
- I have used diagnose debug flow to try to see the traffic arriving at FGT-B with no result
- The client also connects to the server on the same port using a reverse rule; this is working fine.
- On the incoming policy at FGT-B I set auto-asic-offload disable to try to see more traffic
- FGT-B was recently power cycled for other reasons which did not alter the behaviour.
- Both FGT-A and FGT-B are a-p clusters of FGT60E running 5.4.4
- At FGT-A I cleared the existing firewall policy session; a new session was created on the next packet as expected.
- Note, the client is 1-1 nated behind FGT-B using both vip and ippool but (a) that solution works and (b) traffic hasn't even arrived the the external interface yet.
 
I am seeking suggestions to fault find this. I am familiar with di de sni pac and di de flow, and routes are present. Routing looks OK, the policies look OK, the firewall address subnets look OK. I'm a bit baffled.
 
PS. I posted this under VPN because the issue appears to be traffic entered a tunnel but didn't come out. I appreciate the apparent stupidity of that remark and to quote some small people I know "It can't be no-where".
#1

5 Replies Related Threads

    Toshi Esumi
    Expert Member
    • Total Posts : 1259
    • Scores: 89
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: Specific traffic appears to enter IPsec tunnel but not exit!? 2018/11/16 08:36:04 (permalink)
    0
    It sounds like an asic related issue. I'm not particularly familiar with debugging methods with asic but TAC would do it when you open a case. But 5.4.4 is quite old and the latest of 5.4 is now 5.4.10. Have you consider upgrading it? Likely it would at least change the behavior even if it doesn't solve it.
    #2
    ede_pfau
    Expert Member
    • Total Posts : 5751
    • Scores: 397
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Specific traffic appears to enter IPsec tunnel but not exit!? 2018/11/17 15:39:30 (permalink)
    0
    For sniffing traffic, you need to disable traffic offloading to the NP ASIC. Otherwise, you will see the initial session setup and - click! - the trace is dead. This is because the sniffer cannot look into the NP, only into traffic running across the CPU.
    Disable offloading for a specific policy in the CLI:
    config firewall policy
    edit xx
       set auto-asic-offload disable


    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #3
    emnoc
    Expert Member
    • Total Posts : 5082
    • Scores: 311
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Specific traffic appears to enter IPsec tunnel but not exit!? 2018/11/17 19:55:44 (permalink)
    0
    If diag debug flow at FGT-B shows nothing,  I would double check 1> routing a 2> TTL of the SYNs
     

    PCNSE,  NSE , Forcepoint ,  StrongSwan Specialist
    #4
    journeyman
    Gold Member
    • Total Posts : 169
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/03/15 22:56:36
    • Status: offline
    Re: Specific traffic appears to enter IPsec tunnel but not exit!? 2018/11/18 20:57:34 (permalink)
    0
    Thank you for your comments gentlemen.
     
    I have not made much progress on this issue.
     
    @emnoc SYN TTL is 119 entering the ipsec tunnel. If there's a routing issue I cannot see it although I have re-checked. traffic in the reverse direction is fine (client SYN).
     
    @ede auto-asic-offload disable is set on the FGT-B ingress policy. However I just want to see the session established so theoretically I don't need it. The session doesn't establish because we just see the server SYN retry retry retry...
     
    @Toshi, I honestly didn't think of TAC but case logged :)
     
    Thanks again.
    #5
    journeyman
    Gold Member
    • Total Posts : 169
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/03/15 22:56:36
    • Status: offline
    Re: Specific traffic appears to enter IPsec tunnel but not exit!? 2018/11/28 15:06:54 (permalink)
    0
    Well this is interesting.
    FGT-A runs approx 40 similar tunnels. We see the same behaviour on 5 of them. Four failed at the same time, outside of business hours with no known other events (The fifth is FGT-B which was powered down at that time and failed in the same way when it was powered up). The fault would appear to be at FGT-A.
    Working with TAC, we set vpn ipsec phase1-interface npu-offload to disable on the tunnel FGT-A to FGT-B (both ends). This drops the tunnel and when it came back up full functionality was restored.
     
    Unfortunately TAC have gone quiet on the ticket.
     
    I assume that setting npu-offload causes some low level restart of something. By extension, setting npu-offload on the other failed tunnels, or rebooting the entire unit, will "probably" fix it. My preference is to be able to identify the point of failure. It would be very handy to know some diagnostics to identify what is not healthy and therefore how to fix it other than by rebooting. Obviously we don't want to leave npu-offload disabled.
     
    Any suggestions how to identify what is at fault? I'm thinking get sys, diag sys or fnsysctl.
    #6
    Jump to:
    © 2018 APG vNext Commercial Version 5.5