Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
Donnie_Brasco
New Contributor

IPsec: Ping works, TCP fails

Hi all I have a problem with traffic between two FortiGates over IPSec and don't know what to do. I hope you have some ideas that will help me. The problem is that ping works but TCP mostly doesn't - the size of the data (packet size) somehow matters. HTTP/TLS does not work, but SSH (TCP) and Ping do. However, when I transfer large data via SSH (curl, cat, vi), the connection breaks. This leads me to assume that there is a problem with the packet size and MTU/MSS - but I could not confirm this so far. Here is the setup I have - for simplicity, I have adjusted the IPs:

[ol]
  • FTG-40F (Client), Firmware 6.2.7, VDOM (root), PPPoE 111.0.0.5, Public IP 111.0.0.2, Gateway 111.0.0.1
  • FTG-50E (Server), Firmware 6.2.7, VDOM (root) 222.0.0.5, Public IP 222.0.0.2, Gateway 222.0.0.1)[/ol]

    Both FTG are connected via IPSec (example client FTG):

    edit "IPSEC_TUNNEL"
        set interface "wan"
        set keylife 28800
        set peertype any
        set net-device disable
        set proposal aes256-sha256
        set ip-fragmentation pre-encapsulation
        set dhgrp 14
        set nattraversal disable
        set remote-gw 222.0.0.2
        set psksecret ENC AAA...
    next
    As said, pings from both directions are successful. When I submit an HTTP request through the tunnel (requesting a text file with 30KB from the webserver), I can see the request on the web server (HTTP code 200), but no response arrives at the client. If I start a packet capture on both sides of the ISPsec tunnel, I see many TCP retransmissions from the web server in Wireshark - the responses never arrive at the client and a timeout follows. What I checked:

    [ul]
  • Policies are OK
  • routing is OK
  • deactivating all NAT objects
  • changing MTU/MSS (packets are smaller than the maximum MTU according to Wireshark)
  • set ip-fragmentation pre-encapsulation[/ul]

    Diagnose debug flow from client's point of view (FTG-40F) looks like this (response) - possibly there are indications of the problem here, but I don't recognize it:

    2021-01-16 22:43:15 id=20085 trace_id=6112 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 222.0.0.5:8080->111.0.0.5:61252) from IPSEC_TUNNEL. flag [.], seq 4052307548, ack 192939419, win 507"
    2021-01-16 22:43:15 id=20085 trace_id=6112 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000ee9bc, reply direction"
    2021-01-16 22:43:15 id=20085 trace_id=6112 func=npu_handle_session44 line=1159 msg="Trying to offloading session from IPSEC_TUNNEL to lan, skb.npu_flag=00000000 ses.state=00000204 ses.npu_state=0x03040000"
    2021-01-16 22:43:15 id=20085 trace_id=6112 func=fw_forward_dirty_handler line=399 msg="state=00000204, state2=00000001, npu_state=03040000"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 111.0.0.5:61252->222.0.0.5:8080) from lan. flag [.], seq 192939418, ack 4052291108, win 2054"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000ee9bc, original direction"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=npu_handle_session44 line=1159 msg="Trying to offloading session from lan to IPSEC_TUNNEL, skb.npu_flag=00000000 ses.state=00000204 ses.npu_state=0x03040000"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=fw_forward_dirty_handler line=399 msg="state=00000204, state2=00000001, npu_state=03040000"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=ipsecdev_hard_start_xmit line=788 msg="enter IPsec interface-IPSEC_TUNNEL"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=esp_output4 line=927 msg="IPsec encrypt/auth"
    2021-01-16 22:43:25 id=20085 trace_id=6113 func=ipsec_output_finish line=617 msg="send to 111.0.0.1 via intf-ppp1"
    2021-01-16 22:43:25 id=20085 trace_id=6114 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 222.0.0.5:8080->111.0.0.5:61252) from IPSEC_TUNNEL. flag [.], seq 4052307548, ack 192939419, win 507"
    2021-01-16 22:43:25 id=20085 trace_id=6114 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000ee9bc, reply direction"
    2021-01-16 22:43:25 id=20085 trace_id=6114 func=npu_handle_session44 line=1159 msg="Trying to offloading session from IPSEC_TUNNEL to lan, skb.npu_flag=00000000 ses.state=00000204 ses.npu_state=0x03040000"
    2021-01-16 22:43:25 id=20085 trace_id=6114 func=fw_forward_dirty_handler line=399 msg="state=00000204, state2=00000001, npu_state=03040000"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 111.0.0.5:61252->222.0.0.5:8080) from lan. flag [.], seq 192939418, ack 4052291108, win 2054"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000ee9bc, original direction"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=npu_handle_session44 line=1159 msg="Trying to offloading session from lan to IPSEC_TUNNEL, skb.npu_flag=00000000 ses.state=00000204 ses.npu_state=0x03040000"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=fw_forward_dirty_handler line=399 msg="state=00000204, state2=00000001, npu_state=03040000"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=ipsecdev_hard_start_xmit line=788 msg="enter IPsec interface-IPSEC_TUNNEL"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=esp_output4 line=927 msg="IPsec encrypt/auth"
    2021-01-16 22:43:35 id=20085 trace_id=6115 func=ipsec_output_finish line=617 msg="send to 111.0.0.1 via intf-ppp1"
    From the server's point of view (FTG-50E):
    2021-01-16 22:43:16 id=20085 trace_id=2553 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 222.0.0.5:8080->111.0.0.5:61252) from server. flag [.], seq 4052291108, ack 192939419, win 507"
    2021-01-16 22:43:16 id=20085 trace_id=2553 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000876e0, reply direction"
    2021-01-16 22:43:16 id=20085 trace_id=2553 func=ipsecdev_hard_start_xmit line=788 msg="enter IPsec interface-IPSEC_TUNNEL"
    2021-01-16 22:43:16 id=20085 trace_id=2553 func=esp_output4 line=927 msg="IPsec encrypt/auth"
    2021-01-16 22:43:16 id=20085 trace_id=2553 func=ipsec_output_finish line=617 msg="send to 222.0.0.1 via intf-wan1"
    2021-01-16 22:43:25 id=20085 trace_id=2554 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 111.0.0.5:61252->222.0.0.5:8080) from IPSEC_TUNNEL. flag [.], seq 192939418, ack 4052291108, win 2054"
    2021-01-16 22:43:25 id=20085 trace_id=2554 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000876e0, original direction"
    2021-01-16 22:43:25 id=20085 trace_id=2555 func=print_pkt_detail line=5622 msg="vd-root:0 received a packet(proto=6, 222.0.0.5:8080->111.0.0.5:61252) from server. flag [.], seq 4052307548, ack 192939419, win 507"
    2021-01-16 22:43:25 id=20085 trace_id=2555 func=resolve_ip_tuple_fast line=5702 msg="Find an existing session, id-000876e0, reply direction"
    2021-01-16 22:43:25 id=20085 trace_id=2555 func=ipsecdev_hard_start_xmit line=788 msg="enter IPsec interface-IPSEC_TUNNEL"
    2021-01-16 22:43:25 id=20085 trace_id=2555 func=esp_output4 line=927 msg="IPsec encrypt/auth"
    2021-01-16 22:43:25 id=20085 trace_id=2555 func=ipsec_output_finish line=617 msg="send to 222.0.0.1 via intf-wan1"
    If you have ideas or hints for me, I am very grateful. Currently I do not know where to look further. Regards, Donnie

  • 3 REPLIES 3
    emnoc
    Esteemed Contributor III

    Sounds like routing . So if you have a HTTP response where are you seeing that at b4 the fortigate or after the ipsec diagnostics?

     

    Again routing, I would check the route table

    2021-01-16 22:43:25 id=20085 trace_id=2555 func=ipsec_output_finish line=617 msg="send to 222.0.0.1 via intf-wan1"


    get router infor all | grep 222.0.0

    Ken Felix

    PCNSE 

    NSE 

    StrongSwan  

    PCNSE NSE StrongSwan
    Donnie_Brasco

    Hi Ken Felix,

     

    thanks for your input.

     

    S x.0.0.0/24 [10/0] is directly connected, IPSEC_TUNNEL

     

    On both sides (source/destination) routing points into the tunnel. Also, I am able to ping from both sides - even from the subnet behind the firewall. Only with TCP it stucks.

     

    So if you have a HTTP response where are you seeing that at b4 the fortigate or after the ipsec diagnostics?

     

    I'm not sure exactly what you mean. So when I do a packet capture on the FortiGate on the inbound interface (server side), I see the TCP retransmissions, those replies don't arrive on the client side.

     

    Regards,

    Donnie

    emnoc
    Esteemed Contributor III

    You mention you see a 200 response, so tcp-has to be established at this time and the flow show this from the output. Of maybe we misunderstood?

     

     If pings are working, what's the policy that you have ( id # ), and can you add http/https/8080 to that policy and test? And are you routing public address over the ipsec-tunnel ?

     

     

    Ken Felix

     

    PCNSE 

    NSE 

    StrongSwan  

    PCNSE NSE StrongSwan
    Labels
    Top Kudoed Authors