Hot!VPN IPSEC Error Received ESP packet with unknown SPI.

Page: 12 > Showing page 1 of 2
Author
huyhoang8344
New Member
  • Total Posts : 18
  • Scores: 0
  • Reward points: 0
  • Joined: 2013/07/29 03:19:27
  • Status: offline
2014/08/12 22:16:46 (permalink)
0

VPN IPSEC Error Received ESP packet with unknown SPI.

Hi Guys,

I have 2 Tunnel IPSec VPN and both have same error, it happens randomly and when it happen seems like there is no traffic stream in the tunnel even the monitoring say that VPN is up.

I have been looking a lot but no solution so far. any suggestion would be great

Im using Fortigate 100D at my Site, the client site is PA 500

< Message edited by huyhoang8344 -- 8/12/2014 10:17:13 PM >

Attached Image(s)

#1

37 Replies Related Threads

    Istvan Takacs_FTNT
    Silver Member
    • Total Posts : 118
    • Scores: 15
    • Reward points: 0
    • Joined: 2014/08/05 16:14:08
    • Location: Nowhere, OK
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/12 22:34:46 (permalink)
    0
    You can try to run the following in CLI.

    # diagnose debug application ike -1
    # diagnose debug enable

    That would give you a nice long output. When you had enough, disable it;

    # diagnose debug disable

    and have a look if you can find anything strage.

    # diagnose sniffer packet <ipsec interface> " udp and dst port 500"

    can display any communication issue between the initiator and responder.

    If you can keep it running until the next outage, that might report about some error that helps to troubleshoot the issue.

    In the meantime have a look at the other logs. If it randomly gets dropped, that might be the result of unreliable connectivity/interface issues not necessarily on the Fortigate (especially if it thinks that the VPN is up)
    < Message edited by Istvan Takacs -- 8/12/2014 10:36:34 PM >
    #2
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 00:44:41 (permalink)
    0
    Thanks for your respond.

    Did try all those thing you said but still not find anything yet
    Any advise would be appreciated.


    < Message edited by huyhoang8344 -- 8/13/2014 12:48:42 AM >

    Attached Image(s)

    #3
    emnoc
    Expert Member
    • Total Posts : 5769
    • Scores: 375
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: online
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 01:09:19 (permalink)
    0
    Have you match the p2 cfg on the PaloAlto and FGT ? and what version of panos are you running?

    PCNSE 
    NSE 
    StrongSwan  
    #4
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 01:26:04 (permalink)
    0
    Hi emnoc,
    I have check p2 for both ends such as: keylife, encryption, Authentication. They are OK. Using IKE version 1 . i am sorry i but don' t understand what panos is

    Regards,
    Hoang
    #5
    ede_pfau
    Expert Member
    • Total Posts : 6351
    • Scores: 537
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:02:00 (permalink)
    0
    You might be getting these messages because either the idle timeouts on both sides differ, or the PA device does not recognize the keep-alive packets correctly, and so times out.

    Do you have " auto key" or " keepalive" active on the FGT? Phase1 or phase2?

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #6
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:25:59 (permalink)
    0
    i do have " keepalive" on FGT/ phase 2

    i have checked and both sites have the same conf

    No idea what is going on here

    Attached Image(s)

    #7
    ede_pfau
    Expert Member
    • Total Posts : 6351
    • Scores: 537
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:42:01 (permalink)
    0
    I see that you use address names in the Quick Mode selectors. This might not be related but if building a VPN to a non-Fortigate gateway it is best to use plain IP addresses/subnets.

    If you are using Autokey keepalives on the FGT side it might be that the other device ignores these, and idles out. Anyway, I would not be worried too much as long as the tunnel is up when you need it.

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #8
    emnoc
    Expert Member
    • Total Posts : 5769
    • Scores: 375
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: online
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:52:41 (permalink)
    0
    PANOS = PalaAlto Network OS the software that runs the PA.

    A invalid SPIs are most likely in the phase2 so the IKE debug is not going to help;

    these are see when a new SPI switchover or one side expires a SA by byte-sent or seconds before the other from my experience


    Here' s what I would do;


    monitor the ipsec sa ( FGT )

    diag vpn tunnel list name <the tunnel name > | grep spi
    On the PA500 monitor the counters for the tunnels and drops


    show vpn flow tunnel-id <ID>| match spi
    ( to get the current SPIs it should match the fgt in/out from the above commands )

    show counter global filter severity drop
    show counter global filter severity drop aspect tunnel category flow ( look for the bad or wrong SPI counter )


    Also you should monitor the keylife for the SAs ( in & out ) should be almost identical. I think on the PA you can set the timeout to seconds only and not the number of bytes, but I will have to check my PA200 for that.

    PCNSE 
    NSE 
    StrongSwan  
    #9
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:53:25 (permalink)
    0
    Thanks Ede
    The tunnel is up but seem like the traffic can not pass through like, we have SIP trunk between both sides but when this errors come up, 2 PBX can not communicate with each other, i can not even ping the PBX at the other side
    #10
    ede_pfau
    Expert Member
    • Total Posts : 6351
    • Scores: 537
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 02:56:27 (permalink)
    4 (1)
    Any thoughts about the QM selectors?

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #11
    emnoc
    Expert Member
    • Total Posts : 5769
    • Scores: 375
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: online
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 05:05:32 (permalink)
    4 (1)
    The tunnel is up but seem like the traffic can not pass through like, we have SIP trunk between both sides but when this errors come up, 2 PBX can not communicate with each other, i can not even ping the PBX at the other side


    The diag debug flow would be my 1st step


    e.g

    diag debug reset
    diag debug flow filter addr <pbx host or phone>
    diag debug flow show console enable
    diag debug flow trace start 100


    That would get you start in the right direction.



    PCNSE 
    NSE 
    StrongSwan  
    #12
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 19:22:14 (permalink)
    0
    The diag debug flow would be my 1st step


    e.g

    diag debug reset
    diag debug flow filter addr <pbx host or phone>
    diag debug flow show console enable
    diag debug flow trace start 100


    That would get you start in the right direction.


    I got nothing from output. it just happens randomly, don' t know why and when it happens. Thank you
    Any thoughts about the QM selectors


    I have tried and let see it works or not. Thanks you in advance
    Regards,
    Hoang

    < Message edited by huyhoang8344 -- 8/13/2014 7:26:46 PM >

    Attached Image(s)

    #13
    Istvan Takacs_FTNT
    Silver Member
    • Total Posts : 118
    • Scores: 15
    • Reward points: 0
    • Joined: 2014/08/05 16:14:08
    • Location: Nowhere, OK
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 20:12:21 (permalink)
    4 (1)
    you may need to add the following at the end;

    # diagnose debug enable
    #14
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 20:31:54 (permalink)
    0
    id=13 trace_id=739 func=esp_output4 line=885 msg=" encrypting, and send to 203.120.202.66 with source 113.190.252.236"
    id=13 trace_id=739 func=ipsec_output_finish line=231 msg=" send to 123.16.144.1 via intf-ppp1"
    id=13 trace_id=740 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.95.102.70:53->10.171.80.100:51451) from ppp1."
    id=13 trace_id=740 func=resolve_ip_tuple_fast line=4335 msg=" Find an existing session, id-0004e6a4, reply direction"
    id=13 trace_id=740 func=vf_ip4_route_input line=1603 msg=" find a route: gw-10.171.80.100 via Auto"
    id=13 trace_id=740 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=reply)"
    id=13 trace_id=740 func=insert_vlan_header line=53 msg=" insert vlan cos:0 id:9"
    id=13 trace_id=740 func=__if_queue_push_xmit line=364 msg=" send out via dev-port15, dst-mac-00:09:0f:b8:1b:40"
    id=13 trace_id=741 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.171.101.114:62305->10.95.102.70:53) from Wearnes."
    id=13 trace_id=741 func=init_ip_session_common line=4430 msg=" allocate a new session-0004e6f8"
    id=13 trace_id=741 func=vf_ip4_route_input line=1603 msg=" find a route: gw-123.16.144.1 via ppp1"
    id=13 trace_id=741 func=__iprope_tree_check line=534 msg=" use addr/intf hash, len=8"
    id=13 trace_id=741 func=fw_forward_handler line=664 msg=" Allowed by Policy-25: encrypt"
    id=13 trace_id=741 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=original)"
    id=13 trace_id=741 func=ipsec_tunnel_output4 line=818 msg=" enter IPsec tunnel-Tunel_1"
    id=13 trace_id=741 func=esp_output4 line=885 msg=" encrypting, and send to 203.120.202.66 with source 113.190.252.236"
    id=13 trace_id=741 func=ipsec_output_finish line=231 msg=" send to 123.16.144.1 via intf-ppp1"
    id=13 trace_id=742 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.171.101.114:62851->10.95.102.70:53) from Wearnes."
    id=13 trace_id=742 func=init_ip_session_common line=4430 msg=" allocate a new session-0004e6fe"
    id=13 trace_id=742 func=vf_ip4_route_input line=1603 msg=" find a route: gw-123.16.144.1 via ppp1"
    id=13 trace_id=742 func=__iprope_tree_check line=534 msg=" use addr/intf hash, len=8"
    id=13 trace_id=742 func=fw_forward_handler line=664 msg=" Allowed by Policy-25: encrypt"
    id=13 trace_id=742 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=original)"
    id=13 trace_id=742 func=ipsec_tunnel_output4 line=818 msg=" enter IPsec tunnel-Tunel_1"
    id=13 trace_id=742 func=esp_output4 line=885 msg=" encrypting, and send to 203.120.202.66 with source 113.190.252.236"
    id=13 trace_id=742 func=ipsec_output_finish line=231 msg=" send to 123.16.144.1 via intf-ppp1"
    id=13 trace_id=743 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.95.102.70:53->10.171.101.114:62305) from ppp1."
    id=13 trace_id=743 func=resolve_ip_tuple_fast line=4335 msg=" Find an existing session, id-0004e6f8, reply direction"
    id=13 trace_id=743 func=vf_ip4_route_input line=1603 msg=" find a route: gw-10.171.101.114 via Wearnes"
    id=13 trace_id=743 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=reply)"
    id=13 trace_id=743 func=insert_vlan_header line=53 msg=" insert vlan cos:0 id:9"
    id=13 trace_id=743 func=__if_queue_push_xmit line=364 msg=" send out via dev-port15, dst-mac-00:09:0f:b8:1b:40"
    id=13 trace_id=744 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.171.101.114:53123->10.95.102.70:53) from Wearnes."
    id=13 trace_id=744 func=init_ip_session_common line=4430 msg=" allocate a new session-0004e703"
    id=13 trace_id=744 func=vf_ip4_route_input line=1603 msg=" find a route: gw-123.16.144.1 via ppp1"
    id=13 trace_id=744 func=__iprope_tree_check line=534 msg=" use addr/intf hash, len=8"
    id=13 trace_id=744 func=fw_forward_handler line=664 msg=" Allowed by Policy-25: encrypt"
    id=13 trace_id=744 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=original)"
    id=13 trace_id=744 func=ipsec_tunnel_output4 line=818 msg=" enter IPsec tunnel-Tunel_1"
    id=13 trace_id=744 func=esp_output4 line=885 msg=" encrypting, and send to 203.120.202.66 with source 113.190.252.236"
    id=13 trace_id=745 func=resolve_ip_tuple_fast line=4299 msg=" vd-root received a packet(proto=17, 10.171.101.114:55385->10.95.102.70:53) from Wearnes."
    id=13 trace_id=745 func=init_ip_session_common line=4430 msg=" allocate a new session-0004e704"
    id=13 trace_id=745 func=vf_ip4_route_input line=1603 msg=" find a route: gw-123.16.144.1 via ppp1"
    id=13 trace_id=745 func=__iprope_tree_check line=534 msg=" use addr/intf hash, len=8"
    id=13 trace_id=745 func=fw_forward_handler line=664 msg=" Allowed by Policy-25: encrypt"
    id=13 trace_id=745 func=__ip_session_run_tuple line=2558 msg=" run helper-dns-udp(dir=original)"
    id=13 trace_id=745 func=ipsec_tunnel_output4 line=818 msg=" enter IPsec tunnel-Tunel_1"
    id=13 trace_id=745 func=esp_output4 line=885 msg=" encrypting, and send to 203.120.202.66 with source 113.190.252.236"
    id=13 trace_id=744 func=ipsec_output_finish line=231 msg=" send to 123.16.144.1 via intf-ppp1"





    id=13 trace_id=750 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=750 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=750 msg=" syned but no ack, drop"
    id=13 trace_id=751 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=751 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=751 msg=" syned but no ack, drop"
    id=13 trace_id=752 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=752 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=752 msg=" syned but no ack, drop"
    id=13 trace_id=753 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=753 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=753 msg=" syned but no ack, drop"
    id=13 trace_id=754 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=754 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=754 msg=" syned but no ack, drop"
    id=13 trace_id=755 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=755 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=755 msg=" syned but no ack, drop"
    id=13 trace_id=756 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=756 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=756 msg=" syned but no ack, drop"
    id=13 trace_id=757 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=757 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=757 msg=" syned but no ack, drop"
    id=13 trace_id=758 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=758 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=758 msg=" syned but no ack, drop"
    id=13 trace_id=759 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=759 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=759 msg=" syned but no ack, drop"
    id=13 trace_id=760 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=760 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=760 msg=" syned but no ack, drop"
    id=13 trace_id=761 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=761 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=761 msg=" syned but no ack, drop"
    id=13 trace_id=762 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=762 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=762 msg=" syned but no ack, drop"
    id=13 trace_id=763 msg=" vd-root received a packet(proto=6, 10.171.230.8:5060->10.98.230.8:5060) from vlan13."
    id=13 trace_id=763 msg=" Find an existing session, id-000040b1, original direction"
    id=13 trace_id=763 msg=" syned but no ack, drop"

    Here is the output, any suggest would be so great you guys. The VPN tunnel are still up but tracffic can not get through
    < Message edited by huyhoang8344 -- 8/13/2014 8:45:36 PM >
    #15
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/13 20:34:44 (permalink)
    0
    Any thoughts about the QM selectors?

    seems like does not work. Thanks
    #16
    ede_pfau
    Expert Member
    • Total Posts : 6351
    • Scores: 537
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/14 01:48:30 (permalink)
    0
    The second trace shows SIP traffic not completing. Is this traffic across the tunnel? Anyway, this could have many reasons. Mainly, the receiver does not respond, does not want to or is not able to because traffic is blocked. Hard to tell from here.

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #17
    emnoc
    Expert Member
    • Total Posts : 5769
    • Scores: 375
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: online
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/14 02:25:43 (permalink)
    0
    I have to agreed the SIP ( tcp ) is not being ACK.d Are you sure it' s tcp and not 5060/udp or 5061/udp as an alternative?


    On the PA you can execute something similar to the diag debug flow;


    debug dataplane packet-diag set filter match destination x.x.x.x>
    debug dataplane packet-diag set filter match source < y.u.u.u>
    debug dataplane packet-diag set filter on
    debug dataplane packet-diag set log feature flow basic
    debug dataplane packet-diag set log on


    and then clear it when done;


    debug dataplane packet-diag set log off


    As you can see, it' s very fortinet and more juniper SRX like :)


    On the SPI errors is this a policy-based vpn and encrypt action by ID25 ?

    What if you built this as a route-based vpn would the SPI error still be present? If your only complaint is that of the invalid SPI, than I would not worry to much.

    For the QM proxy-ids, they need to match what the PA500 has, Do you have access to the PA? Did you get any of the output that was suggested? and mainly the wrong SPI ?

    Can you get the vpn tunnel statius via ?


    show vpn ike-sa ( phase1 related goodies )
    show vpn ipsec-sa ( phase2 related goodies )


    Once again very SRX like.

    PCNSE 
    NSE 
    StrongSwan  
    #18
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/14 03:23:18 (permalink)
    0
    I have configured policy-based VPN and i have been searching around that route-based has the same issue and still not get fixed yet and the rule between 2 PBX is allowed for all Services and this traffic is across the tunnel so TCP or UDP should be OK
    Both QM proxy-IDs are matched, that is why the tunnel is up and working fine until the errors came
    I do not have access to PA500 and all the output which was posted here and that is all i got so far..


    #19
    huyhoang8344
    New Member
    • Total Posts : 18
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/07/29 03:19:27
    • Status: offline
    RE: VPN IPSEC Error Received ESP packet with unknown SPI. 2014/08/18 06:23:24 (permalink)
    0
    First i want to thank emnoc and ede_pfau. based on your advices i have just fixed this issue apparently.
    Second, i want to update a little bit how i fix it. As said i do not have access to PA 500 so i do not know what kind of VPN configuration that device have so i was using policy based VPN which is easier than route-based VPN and the problem happens on and on randomly. Now i have changed to route-based VPN then there is no errors messages anymore. Seems like PA500 is configured as route based VPN

    Thanks all of you
    #20
    Page: 12 > Showing page 1 of 2
    Jump to:
    © 2020 APG vNext Commercial Version 5.5