Hot!Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back.

Author
cgbaroudi
New Member
  • Total Posts : 9
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/09/14 16:09:20
  • Status: offline
2017/10/12 06:32:00 (permalink)
0

Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back.

Hi,
 
I have 2 VPNs UP with AWS VPC. The VPNs are configured with failover so if one goes down the other picks up. Below we can see one VPN is up:
 
34.232.188.255 169.254.47.44/30 DOWN52.86.222.130 169.254.44.184/30 UP 
 
Info:


1. When I ping from my Fortigate to the EC2, I can see that the EC2 receives the ping request and sends a reply. The reply however never makes it back through the tunnel:tcpdump output on the AWS EC2:12:50:58.923392 IP 169.254.44.186 > 192.168.241.142: ICMP echo request, id 2769612:50:58.923405 IP 192.168.241.142 > 169.254.44.186: ICMP echo reply, id 276963.

2. From the fortigate I can ping both ends of the tunnel which is obvious from the time in ms:
execute ping 169.254.44.185
PING 169.254.44.185 (169.254.44.185): 56 data bytes64 bytes from 169.254.44.185: icmp_seq=0 ttl=254 time=13.0 ms64 bytes from 169.254.44.185: icmp_seq=1 ttl=254 time=13.1 ms
execute ping 169.254.44.186
PING 169.254.44.186 (169.254.44.186): 56 data bytes64 bytes from 169.254.44.186: icmp_seq=0 ttl=255 time=0.0 ms64 bytes from 169.254.44.186: icmp_seq=1 ttl=255 time=0.0 ms4.

3. From the EC2 I cannot ping either end of the tunnel.5. SSH fails in the same way.

4. We see ssh session attempts and replies which do not make it back from the EC2:
 
06:00:30.829303 IP 192.168.241.142.ssh > 169.254.44.186.9589: Flags [S.], seq 2689521519, ack 2166390256, win 26847, options [mss 8961,sackOK,TS val 4354044 ecr 236943444,nop,wscale 7], length 0
06:00:38.633577 IP 169.254.44.186.9589 > 192.168.241.142.ssh: Flags , seq 2166390255, win 5272, options [mss 1318,sackOK,TS val 236945544 ecr 0,nop,wscale 2], length 0


Seems that traffic from the EC2 back to the Fortigate is being blocked yet debug flow on the fortigate shows no attempts or denial:


# execute ssh 192.168.241.142
id=20085 trace_id=2273 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. flag , seq 2166390255, ack 0, win 5272"
id=20085 trace_id=2273 func=init_ip_session_common line=5449 msg="allocate a new session-01a8007c"
id=20085 trace_id=2273 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]"
id=20085 trace_id=2273 func=iprope_dnat_tree_check line=835 msg="len=0"
id=20085 trace_id=2273 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
 
Any ideas would be very helpful. Please!!!
 
Thank you and Best Regards.
 
 



post edited by cgbaroudi - 2017/10/12 06:40:08
#1

9 Replies Related Threads

    Paul S
    Gold Member
    • Total Posts : 154
    • Scores: 6
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 09:03:39 (permalink)
    0
    Maybe a mismatch of IPSEC encryption domain? NAT issue?  Maybe we need more info about your setup.
     
    is this line related to the VPN?
    id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    is 192.168.241.142 behind WAN1?
     
    "enter IPsec interface-vpn-23766442-1"
    shouldn't the action be leaving the tunnel?

    FG200D 5.2.10 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.4.2  |  Fortimail 5.3.8
    Network+, Security+
    #2
    cgbaroudi
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/09/14 16:09:20
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 09:48:43 (permalink)
    0
    Hi Paul,


    Thank you for the response. I will answer your questions first then follow up with the complete CLI configuration.
     
    1. 192.168.241.142 is the AWS VPC subnet


    2. In the log you referred to I have "diag flow" enabled and filtering on one of the VPN endpoints (169.254.44.186 & 185)


    3. From the log below, I understand that the fortigate figured that the ssh command to  192.168.241.142  is to be routed to the VPN interface vpn-23766442-1 then it says sending through the physical default route of  66.162.199.129 thru wan1. Am I correct? The follow up post has the full diag flow trace logs for ping and ssh.
     
    Log:
    id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
     
    Complete CLI config on fortigate follows. (version 5.6.2). Please note I made changes to update the setting to work with v5.6.2:
     
    ! Amazon Web Services
    ! Virtual Private Cloud
    ! AWS utilizes unique identifiers to manipulate the configuration of
    ! a VPN Connection. Each VPN Connection is assigned an identifier and is
    ! associated with two other identifiers, namely the
    ! Customer Gateway Identifier and Virtual Private Gateway Identifier.
    !
    ! Your VPN Connection ID : vpn-23766442
    ! Your Virtual Private Gateway ID : vgw-28da3041
    ! Your Customer Gateway ID : cgw-3720ca5e
    !
    !
    ! This configuration consists of two tunnels. Both tunnels must be
    ! configured on your Customer Gateway.
    !
    ! --------------------------------------------------------------------------------
    ! IPSec Tunnel #1
    ! --------------------------------------------------------------------------------
    ! #1: Internet Key Exchange (IKE) Configuration
    !
    ! A policy is established for the supported ISAKMP encryption,
    ! authentication, Diffie-Hellman, lifetime, and key parameters.
    ! Please note, these sample configurations are for the minimum requirement of AES128, SHA1, and DH Group 2.
    ! You will need to modify these sample configuration files to take advantage of AES256, SHA256, or other DH groups like 2, 14-18, 22, 23, and 24.
    !
    ! The address of the external interface for your customer gateway must be a static address.
    ! Your customer gateway may reside behind a device performing network address translation (NAT).
    ! To ensure that NAT traversal (NAT-T) can function, you must adjust your firewall rules to unblock UDP port 4500. If not behind NAT, we recommend disabling NAT-T.
    !
    ! Configuration begins in root VDOM.


    config vpn ipsec phase1-interface
    edit vpn-23766442-0 ! Name must be shorter than 15 chars, best if shorter than 12
    set interface "wan1"
    ! The IPSec Dead Peer Detection causes periodic messages to be
    ! sent to ensure a Security Association remains operational
    set dpd enable
    set local-gw 66.x.x.x
    set dhgrp 2
    set proposal aes128-sha1
    set keylife 28800
    set remote-gw 34.232.188.255
    set psksecret pV5rbsXlyhWYLWEXB.gkmS56nMktjEMB
    set dpd-retryinterval 10
    next
    end

    ! #2: IPSec Configuration
    !
    ! The IPSec transform set defines the encryption, authentication, and IPSec
    ! mode parameters.
    !
    ! Please note, you may use these additionally supported IPSec parameters for encryption like AES256 and other DH groups like 2, 5, 14-18, 22, 23, and 24.
     
    config vpn ipsec phase2-interface
    edit "vpn-23766442-0"
    set phase1name "vpn-23766442-0"
    set proposal aes128-sha1
    set dhgrp 2
    set pfs enable
    set keylifeseconds 3600
    next
     
    ! --------------------------------------------------------------------------------
    ! #3: Tunnel Interface Configuration
    !
    ! A tunnel interface is configured to be the logical interface associated
    ! with the tunnel. All traffic routed to the tunnel interface will be
    ! encrypted and transmitted to the VPC. Similarly, traffic from the VPC
    ! will be logically received on this interface.
    !
    !
    ! The address of the interface is configured with the setup for your
    ! Customer Gateway. If the address changes, the Customer Gateway and VPN
    ! Connection must be recreated with Amazon VPC.
    ! Perform this from the Global VDOM.
    config global
    config system interface
    edit "vpn-23766442-0"
    set vdom "root"
    set ip 169.254.47.46 255.255.255.255
    set allowaccess ping
    set type tunnel
    ! This option causes the router to reduce the Maximum Segment Size of
    ! TCP packets to prevent packet fragmentation.
    !
    set tcp-mss 1379
    set remote-ip 169.254.47.45
    ??? set mtu 1427
    set interface "wan1"
    next

    ! ----------------------------------------------------------------------------
    ! #4 Static Route Configuration
    !
    ! Your Customer Gateway needs to set a static route for the prefix corresponding to your
    ! VPC to send traffic over the tunnel interface.
    ! An example for a VPC with the prefix 10.0.0.0/16 is provided below:
    !
    ! Configuration begins in root VDOM.
    ! First obtain the numbered list of static routes in use:
    config vdom
    edit root
    show router static
    ! This will have entries for edit 1, edit 2, etc., showing the device, destination, and gateway for each static route configured. You need to use an unused routing entry for the static route into your VPC. So for example if there were routes 1, 2, and 3, then you would edit 4 below:
    config router static
    edit 4
    set device "vpn-23766442-0"
    set dst 192.168.241.142 255.255.0.0
    next
    end
    ! Static routing does not allow for failover of traffic between tunnels. If there is a problem with one of the
    ! tunnels, we would want to failover the traffic to the second tunnel. This is done by using "gwdetect" in fortigate.
    ! The gwdetect command will ping the other end of the tunnel to check if the tunnel is up. If the pings fail, it will
    ! remove the static route from the routing table, and the second route in the table will become active.
    !
    ! The following config will tell the Fortigate device what IP it should ping to test the tunnel. This IP should be
    ! the inside IP address of the virtual private gateway.
     
    FMU-FW-02 # config system link-monitor
    FMU-FW-02 (link-monitor) # edit 1
    new entry '1' added
    FMU-FW-02 (1) # set srcintf "vpn-23766442-0"
    FMU-FW-02 (1) # set server "169.254.47.45"
    FMU-FW-02 (1) # set interval 2
    FMU-FW-02 (1) # set failtime 5
    FMU-FW-02 (1) # next
    FMU-FW-02 (link-monitor) # end
    FMU-FW-02 # config system link-monitor
    FMU-FW-02 (link-monitor) # show
    config system link-monitor
    edit "1"
    set srcintf "vpn-23766442-0"
    set server "169.254.47.45"
    set interval 2
    next
    end



    ! ----------------------------------------------------------------------------
    ! #5 Firewall Policy Configuration
    !
    ! Create a firewall policy permitting traffic from your local subnet to the VPC subnet and vice versa
    !
    ! This example policy permits all traffic from the local subnet to the VPC
    ! First, find the policies that exist
    ! Configuration begins in root VDOM.
    config vdom
    edit root
    show firewall policy
    ! Next, create a new firewall policy starting with the next available policy ID. If policies 1, 2, 3, and 4 were shown, then in this example the policy created starts at 5
    config firewall policy
    edit 5
    set srcintf "vpn-23766442-0"
    set dstintf "internal" ! This is the interface out which your local LAN resides
    set srcaddr all
    set dstaddr all
    set action accept
    set schedule always
    set service ALL
    next
    end
    ! Now create a policy to permit traffic going the other way
    config firewall policy
    edit 6
    set srcintf internal ! This is the interface out which your local LAN resides
    set dstintf "vpn-23766442-0"
    set srcaddr all
    set dstaddr all
    set action accept
    set schedule always
    set service ALL
    next
    end
    ! --------------------------------------------------------------------------------
    ! IPSec Tunnel #2
    ! --------------------------------------------------------------------------------
    ! #1: Internet Key Exchange (IKE) Configuration
    !
    ! A policy is established for the supported ISAKMP encryption,
    ! authentication, Diffie-Hellman, lifetime, and key parameters.
    ! Please note, these sample configurations are for the minimum requirement of AES128, SHA1, and DH Group 2.
    ! You will need to modify these sample configuration files to take advantage of AES256, SHA256, or other DH groups like 2, 14-18, 22, 23, and 24.
    !
    ! The address of the external interface for your customer gateway must be a static address.
    ! Your customer gateway may reside behind a device performing network address translation (NAT).
    ! To ensure that NAT traversal (NAT-T) can function, you must adjust your firewall rules to unblock UDP port 4500. If not behind NAT, we recommend disabling NAT-T.
    !
    ! Configuration begins in root VDOM.
    config vpn ipsec phase1-interface
    edit vpn-23766442-1 ! Name must be shorter than 15 chars, best if shorter than 12
    set interface "wan1"
    ! The IPSec Dead Peer Detection causes periodic messages to be
    ! sent to ensure a Security Association remains operational
    set dpd enable
    ### set dpd on-demand
    set local-gw 66.x.x.x
    set dhgrp 2
    set proposal aes128-sha1
    set keylife 28800
    set remote-gw 52.86.222.130
    set psksecret k3Z8Q8K1L3MUClD1MbaPQNNSrffFRS5K
    set dpd-retryinterval 10
    next
    ### end

    ! #2: IPSec Configuration
    !
    ! The IPSec transform set defines the encryption, authentication, and IPSec
    ! mode parameters.
    !
    ! Please note, you may use these additionally supported IPSec parameters for encryption like AES256 and other DH groups like 2, 5, 14-18, 22, 23, and 24.
    config vpn ipsec phase2-interface
    edit "vpn-23766442-1"
    set phase1name "vpn-23766442-1"
    set proposal aes128-sha1
    set dhgrp 2
    set pfs enable
    set keylifeseconds 3600
    next
    end
    ! --------------------------------------------------------------------------------
    ! #3: Tunnel Interface Configuration
    !
    ! A tunnel interface is configured to be the logical interface associated
    ! with the tunnel. All traffic routed to the tunnel interface will be
    ! encrypted and transmitted to the VPC. Similarly, traffic from the VPC
    ! will be logically received on this interface.
    !
    !
    ! The address of the interface is configured with the setup for your
    ! Customer Gateway. If the address changes, the Customer Gateway and VPN
    ! Connection must be recreated with Amazon VPC.
    ! Perform this from the Global VDOM.
    config global
    config system interface
    edit "vpn-23766442-1"
    set vdom "root"
    set ip 169.254.44.186 255.255.255.255
    set allowaccess ping
    set type tunnel
    ! This option causes the router to reduce the Maximum Segment Size of
    ! TCP packets to prevent packet fragmentation.
    !
    set tcp-mss 1379
    set remote-ip 169.254.44.185
    ??? set mtu 1427
    set interface "wan1"
    next
    ### endconfig

    ! ----------------------------------------------------------------------------
    ! #4 Static Route Configuration
    !
    ! Your Customer Gateway needs to set a static route for the prefix corresponding to your
    ! VPC to send traffic over the tunnel interface.
    ! An example for a VPC with the prefix 10.0.0.0/16 is provided below:
    !
    ! Configuration begins in root VDOM.
    ! First obtain the numbered list of static routes in use:
    ### below is only for listing static routes. Syntax does not work on v6.3
    config vdom
    edit root
    show router static
    ! This will have entries for edit 1, edit 2, etc., showing the device, destination, and gateway for each static route configured. You need to use an unused routing entry for the static route into your VPC. So for example if there were routes 1, 2, and 3, then you would edit 4 below:
    config router static
    edit 4
    set device "vpn-23766442-1"
    set dst 192.168.0.0 255.255.0.0
    next
    end
    ! Static routing does not allow for failover of traffic between tunnels. If there is a problem with one of the
    ! tunnels, we would want to failover the traffic to the second tunnel. This is done by using "gwdetect" in fortigate.
    ! The gwdetect command will ping the other end of the tunnel to check if the tunnel is up. If the pings fail, it will
    ! remove the static route from the routing table, and the second route in the table will become active.
    !
    ! The following config will tell the Fortigate device what IP it should ping to test the tunnel. This IP should be
    ! the inside IP address of the virtual private gateway.
    ### Version 6.3:
    FMU-FW-02 # config system link-monitor
    FMU-FW-02 (link-monitor) # edit 2
    new entry '1' added
    FMU-FW-02 (1) # set srcintf "vpn-23766442-1"
    FMU-FW-02 (1) # set server "169.254.44.185"
    FMU-FW-02 (1) # set interval 2
    FMU-FW-02 (1) # set failtime 5
    FMU-FW-02 (1) # next
    FMU-FW-02 (link-monitor) # end
    ### Version 5.x
    config vdom
    edit root
    config router gwdetect
    edit 1
    set interface "vpn-23766442-1"
    set server "169.254.44.185"
    ! Using the following command, you can set the interval and failtime for gwdetect. Interval is number of seconds
    ! between pings. Failtime is the number of lost consecutive pings.Using the respective values of 2 and 5, your tunnel
    ! will failover in 10 seconds.
    !
    set interval 2
    set failtime 5
    next
    end
    ! ----------------------------------------------------------------------------
    ! #5 Firewall Policy Configuration
    !
    ! Create a firewall policy permitting traffic from your local subnet to the VPC subnet and vice versa
    !
    ! This example policy permits all traffic from the local subnet to the VPC
    ! First, find the policies that exist
    ! Configuration begins in root VDOM.
    config vdom
    edit root
    show firewall policy
    ! Next, create a new firewall policy starting with the next available policy ID. If policies 1, 2, 3, and 4 were shown, then in this example the policy created starts at 5
    config firewall policy
    edit 5
    set srcintf "vpn-23766442-1"
    set dstintf "internal" ! This is the interface out which your local LAN resides
    set srcaddr all
    set dstaddr all
    set action accept
    set schedule always
    set service ALL
    next
    end
    ! Now create a policy to permit traffic going the other way
    config firewall policy
    edit 6
    set srcintf internal ! This is the interface out which your local LAN resides
    set dstintf "vpn-23766442-1"
    set srcaddr all
    set dstaddr all
    set action accept
    set schedule always
    set service ALL
    next
    end
     
     
     
     
    post edited by cgbaroudi - 2017/10/12 13:03:48
    #3
    cgbaroudi
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/09/14 16:09:20
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 12:37:16 (permalink)
    0
    Here are the full flow trace logs for ping and ssh from the local Fortigate to the AWS EC2 (Centos):
     
    FMU-FW-02 # diag debug flow filter addr 192.168.241.142
    FMU-FW-02 # diag debug flow trace start 100
    FMU-FW-02 # execute ssh 192.168.241.142
    id=20085 trace_id=2273 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl
    id=20085 trace_id=2273 func=init_ip_session_common line=5449 msg="allocate a new session-01a8007c"
    id=20085 trace_id=2273 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]"
    id=20085 trace_id=2273 func=iprope_dnat_tree_check line=835 msg="len=0"
    id=20085 trace_id=2273 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000"
    id=20085 trace_id=2273 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2273 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2273 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2274 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl
    id=20085 trace_id=2274 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a8007c, original direction"
    id=20085 trace_id=2274 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2274 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2274 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2275 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl
    id=20085 trace_id=2275 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a8007c, original direction"
    id=20085 trace_id=2275 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2275 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2275 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2276 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl
    id=20085 trace_id=2276 func=init_ip_session_common line=5449 msg="allocate a new session-01a8018e"
    id=20085 trace_id=2276 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]"
    id=20085 trace_id=2276 func=iprope_dnat_tree_check line=835 msg="len=0"
    id=20085 trace_id=2276 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000"
    id=20085 trace_id=2276 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2276 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2276 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2277 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=6, 169.254.44.186:9589->192.168.241.142:22) from local. fl
    id=20085 trace_id=2277 func=init_ip_session_common line=5449 msg="allocate a new session-01a802e0"
    id=20085 trace_id=2277 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]"
    id=20085 trace_id=2277 func=iprope_dnat_tree_check line=835 msg="len=0"
    id=20085 trace_id=2277 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000"
    id=20085 trace_id=2277 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2277 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2277 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
     
     
    FMU-FW-02 # execute ping 192.168.241.142
    PING 192.168.241.142 (192.168.241.142): 56 data bytes
    id=20085 trace_id=2278 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local.
    id=20085 trace_id=2278 func=init_ip_session_common line=5449 msg="allocate a new session-01a83a1d"
    id=20085 trace_id=2278 func=iprope_dnat_check line=4754 msg="in-[], out-[vpn-23766442-1]"
    id=20085 trace_id=2278 func=iprope_dnat_tree_check line=835 msg="len=0"
    id=20085 trace_id=2278 func=iprope_dnat_check line=4767 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000"
    id=20085 trace_id=2278 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2278 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2278 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2279 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local.
    id=20085 trace_id=2279 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction"
    id=20085 trace_id=2279 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2279 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2279 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2280 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local.
    id=20085 trace_id=2280 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction"
    id=20085 trace_id=2280 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2280 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2280 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2281 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local.
    id=20085 trace_id=2281 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction"
    id=20085 trace_id=2281 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2281 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2281 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
    id=20085 trace_id=2282 func=print_pkt_detail line=5293 msg="vd-root received a packet(proto=1, 169.254.44.186:47277->192.168.241.142:2048) from local.
    id=20085 trace_id=2282 func=resolve_ip_tuple_fast line=5368 msg="Find an existing session, id-01a83a1d, original direction"
    id=20085 trace_id=2282 func=ipsecdev_hard_start_xmit line=144 msg="enter IPsec interface-vpn-23766442-1"
    id=20085 trace_id=2282 func=esp_output4 line=1174 msg="IPsec encrypt/auth"
    id=20085 trace_id=2282 func=ipsec_output_finish line=534 msg="send to 66.162.199.129 via intf-wan1"
     
    --- 192.168.241.142 ping statistics ---
    5 packets transmitted, 0 packets received, 100% packet loss
     
    #4
    emnoc
    Expert Member
    • Total Posts : 4400
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 12:40:16 (permalink)
    0
    Sugestion
     
    1>
    if you have 2x VPN shutting one down and try your test
     
    2>
    bring the one shutdown back up and shutdown the other and retest
     
    3>
    run diag sniffer packet <vpn interface name> "icmp" while doing your ping test
     
    4>
    Dump the route table for the  VPN-tunnels
     
    e.g
     
    get router info routing all | gre vpn
     
    Ken

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #5
    cgbaroudi
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/09/14 16:09:20
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 14:09:54 (permalink)
    0
    Hi Ken,
     
    Thank you.

    1> if you have 2x VPN shutting one down and try your test
    Answer: One VPN was already shut down.
     
    2> bring the one shutdown back up and shutdown the other and retest
    Answer: Done and got the same results.
     
    3> run diag sniffer packet <vpn interface name> "icmp" while doing your ping test
    Answer: The log from the diag sniffer command provided below. Note the heartbeat pings before and after the ping test. No replies from 192.168.241.142:


    5.684973 169.254.47.46 -> 169.254.47.45: icmp: echo request
    5.696837 169.254.47.45 -> 169.254.47.46: icmp: echo reply
    5.828415 169.254.47.46 -> 192.168.241.142: icmp: echo request
    6.828322 169.254.47.46 -> 192.168.241.142: icmp: echo request
    7.694965 169.254.47.46 -> 169.254.47.45: icmp: echo request
    7.706828 169.254.47.45 -> 169.254.47.46: icmp: echo reply
    7.828316 169.254.47.46 -> 192.168.241.142: icmp: echo request
    8.828312 169.254.47.46 -> 192.168.241.142: icmp: echo request
    9.704955 169.254.47.46 -> 169.254.47.45: icmp: echo request
    9.716750 169.254.47.45 -> 169.254.47.46: icmp: echo reply
    9.828297 169.254.47.46 -> 192.168.241.142: icmp: echo request
    11.714945 169.254.47.46 -> 169.254.47.45: icmp: echo request
    11.726765 169.254.47.45 -> 169.254.47.46: icmp: echo reply

     
    4> Dump the route table for the  VPN-tunnels
    # get router info routing-table all | grep vpn
    S 10.108.0.0/16 [10/0] via 169.254.34.113, vpn-2574934c-0
    S 10.134.8.0/21 [10/0] via 169.254.14.161, vpn-f95d47eb-0
    C 169.254.14.53/32 is directly connected, vpn-f95d47eb-1
    C 169.254.14.54/32 is directly connected, vpn-f95d47eb-1
    C 169.254.14.161/32 is directly connected, vpn-f95d47eb-0
    C 169.254.14.162/32 is directly connected, vpn-f95d47eb-0
    C 169.254.34.17/32 is directly connected, vpn-2574934c-1
    C 169.254.34.18/32 is directly connected, vpn-2574934c-1
    C 169.254.34.113/32 is directly connected, vpn-2574934c-0
    C 169.254.34.114/32 is directly connected, vpn-2574934c-0
    C 169.254.47.45/32 is directly connected, vpn-23766442-0
    C 169.254.47.46/32 is directly connected, vpn-23766442-0
    S 192.168.0.0/16 [10/0] via 169.254.47.45, vpn-23766442-0
     
    Thank you again.
     
    Kind Regards.
    #6
    Paul S
    Gold Member
    • Total Posts : 154
    • Scores: 6
    • Reward points: 0
    • Joined: 2011/05/02 16:49:52
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 15:18:30 (permalink)
    0
    these route table entries are the only one's for the VPN in question, yes? (vpn-23766442)?
    C 169.254.47.45/32 is directly connected, vpn-23766442-0
    C 169.254.47.46/32 is directly connected, vpn-23766442-0
    S 192.168.0.0/16 [10/0] via 169.254.47.45, vpn-23766442-0
     
    can you ping anything on the 192.168.0.0/16 network since that is on the far end of your VPN?
     
    in your first post you said you see the ping reply, but it doesn't leave the tunnel. That means you see it on one interface, but it never leaves out another interface. what interface is the ping reply supposed to exit on?

    FG200D 5.2.10 (HA) - primary
    FWF50B' s 4.3.x, FG60D's 5.2.x, FG60E's 5.4.x                   [Did my post help you? Please rate my post.]
    FAZ-VM 5.4.2  |  Fortimail 5.3.8
    Network+, Security+
    #7
    cgbaroudi
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/09/14 16:09:20
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/12 21:26:02 (permalink)
    0
    Hi Paul,
    Thank you.
    Correct, the 3 routes you listed are the ones for the VPN in question.
    I will rephrase perhaps more accurately. On the EC2 inside the AWS network I have tcpdump running. When I ping the AWS EC2 from the fortigate or even a PC from our local network, tcpdump will show the ICMP packet coming through the VPN to the EC2, then shows the EC2 responding but the reply never makes it back through the VPN. This also correlates with the fact that I can ping both endpoints of the VPN from the fortigate but I cannot ping either endpoint of the VPN from the AWS EC2 instance which incidentally is running Centos. It appears that something on the fortigate is blocking incoming traffic through the VPN interface while allowing outgoing traffic from the fortigate.
     
    So to answer your second question, no I cannot ping the AWS instance from my local network from the fortigate or host. tcpdump shows me the ping is reaching the AWS EC2, the EC2 replies but the reply cannot get back through the VPN. Same with ssh traffic or any other traffic.
     
    Can we trace dropped incoming VPN traffic? If we can and determine that the fortigate is not dropping incoming traffic then we can focus on AWS causing the issue.
     
    Thank you again. I am very grateful for all your help.
     
    post edited by cgbaroudi - 2017/10/12 21:32:00
    #8
    cgbaroudi
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/09/14 16:09:20
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/13 14:45:34 (permalink)
    0
    Hey guys,
     
    Thank you all for your help.
     
    The problem was as I suspected, static routes needed to be added to the VPN configuration which include:


    a. static route for the VPN tunnel endpoints
    b. static route for all our internal networks
     
    Specifically, in my case, under AWS>VPC>VPN Connections>Static Routes Tab, I added:
     
    VPN endpoints:            169.254.47.44/30
    Internal network 1:      10.20.10.0/24
    Internal network 2:      10.96.0.0/16

    Naturally, I can now ping the tunnel endpoint from the AWS EC2 instance.
     
    Thank you all again (Paul S. and emnoc)
     
    Kind Regards,
    cgb
     
    #9
    emnoc
    Expert Member
    • Total Posts : 4400
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Fortigate 200D (v5.6.2) IPSec VPN to AWS. Traffic makes it to EC2 but not back. 2017/10/13 20:08:58 (permalink)
    0
    I'm glad it all worked out for you in  the end.
     
    Ken
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #10
    Jump to:
    © 2017 APG vNext Commercial Version 5.5