Specific traffic appears to enter IPsec tunnel but not exit!?

[journeyman]

On an ipsec interface tunnel between FGT-A and FGT-B, I see specific traffic (Server SYN) enter the ipsec interface at A but cannot see it exit at B (using diagnose sniffer packet and diagnose debug flow at each end).

The configuration and functionality is not new and is duplicated widely across our system; I see the problem only on one instance.

- The ipsec tunnel also carries OSPF and all necessary routes are visible;

- The ipsec tunnel carries all traffic to and from FGT-B;

- The Client also establishes a session in the reverse direction, this is successful.

- I see no other functionality impacted on the tunnel

- I have checked firewall address and policy configurations

- I have used diagnose debug flow to try to see the traffic arriving at FGT-B with no result

- The client also connects to the server on the same port using a reverse rule; this is working fine.

- On the incoming policy at FGT-B I set auto-asic-offload disable to try to see more traffic

- FGT-B was recently power cycled for other reasons which did not alter the behaviour.

- Both FGT-A and FGT-B are a-p clusters of FGT60E running 5.4.4

- At FGT-A I cleared the existing firewall policy session; a new session was created on the next packet as expected.

- Note, the client is 1-1 nated behind FGT-B using both vip and ippool but (a) that solution works and (b) traffic hasn't even arrived the the external interface yet.

I am seeking suggestions to fault find this. I am familiar with di de sni pac and di de flow, and routes are present. Routing looks OK, the policies look OK, the firewall address subnets look OK. I'm a bit baffled.

PS. I posted this under VPN because the issue appears to be traffic entered a tunnel but didn't come out. I appreciate the apparent stupidity of that remark and to quote some small people I know "It can't be no-where".

[Toshi_Esumi]

It sounds like an asic related issue. I'm not particularly familiar with debugging methods with asic but TAC would do it when you open a case. But 5.4.4 is quite old and the latest of 5.4 is now 5.4.10. Have you consider upgrading it? Likely it would at least change the behavior even if it doesn't solve it.

[ede_pfau]

For sniffing traffic, you need to disable traffic offloading to the NP ASIC. Otherwise, you will see the initial session setup and - click! - the trace is dead. This is because the sniffer cannot look into the NP, only into traffic running across the CPU.

Disable offloading for a specific policy in the CLI:

config firewall policy

edit xx

   set auto-asic-offload disable

[emnoc]

If diag debug flow at FGT-B shows nothing,  I would double check 1> routing a 2> TTL of the SYNs

[journeyman]

Thank you for your comments gentlemen.

I have not made much progress on this issue.

@emnoc SYN TTL is 119 entering the ipsec tunnel. If there's a routing issue I cannot see it although I have re-checked. traffic in the reverse direction is fine (client SYN).

@ede auto-asic-offload disable is set on the FGT-B ingress policy. However I just want to see the session established so theoretically I don't need it. The session doesn't establish because we just see the server SYN retry retry retry...

@Toshi, I honestly didn't think of TAC but case logged :)

Thanks again.

[journeyman]

Well this is interesting.

FGT-A runs approx 40 similar tunnels. We see the same behaviour on 5 of them. Four failed at the same time, outside of business hours with no known other events (The fifth is FGT-B which was powered down at that time and failed in the same way when it was powered up). The fault would appear to be at FGT-A.

Working with TAC, we set vpn ipsec phase1-interface npu-offload to disable on the tunnel FGT-A to FGT-B (both ends). This drops the tunnel and when it came back up full functionality was restored.

Unfortunately TAC have gone quiet on the ticket.

I assume that setting npu-offload causes some low level restart of something. By extension, setting npu-offload on the other failed tunnels, or rebooting the entire unit, will "probably" fix it. My preference is to be able to identify the point of failure. It would be very handy to know some diagnostics to identify what is not healthy and therefore how to fix it other than by rebooting. Obviously we don't want to leave npu-offload disabled.

Any suggestions how to identify what is at fault? I'm thinking get sys, diag sys or fnsysctl.

[ii_networker]

hi journeyman,

i've had a similar issue on one of my vpn tunnels. Sniffer said traffic was going out on vpn interface on one side but never came out on the other side. Interfaces were up. In my case it was a mismatch with local/remote subnets in phase2 config. Dont know exactly which debug i used, but i think it was diagnose debug flow on first side (traffic out), which said traffic was discarded because of this, although sniffer said it was sent out

maybe you have the same problem?

best regards

[btp]

I see that this is an old thread. However - for anyone having the same issue: I had the same problem, and was struggling to fix it. In my case the traffic not getting through was marked with COS - and one of the switches (Juniper) did not have a COS profile associated with the aggregated-ethernet port. Juniper then discarded this traffic. Applying a COS profile solved it.