ADVPN - Only one tunnel works

[noother10]

I'm building a proof of concept to get more understanding around ADVPN and BGP before a future project. I have 3x 100E each with its own internet connection.

I've looked a numerous cookbooks, guides and topics on it. I've managed to configure and build a 1x Hub and 2x Spoke setup. Both Spokes connect via IPSEC tunnel, but only the first connected Spoke can actually do anything, like ping the Hub interface and use BGP. The second Spoke while connected, cannot actually get anywhere or do anything.

Each phase2 on the Spokes is wildcarded (0.0.0.0/0.0.0.0). When I try to ping the hub tunnel interface from the non-functioning spoke interface, it enters the tunnel, I can see it come up in flow trace on the hub but it doesn't seem to get back to the spoke.

id=20085 trace_id=1 func=init_ip_session_common line=5451 msg="allocate a new session-00000340" id=20085 trace_id=1 func=vf_ip_route_input_common line=2576 msg="find a route: flag=80000000 gw-10.0.10.1 via root" id=20085 trace_id=2 func=print_pkt_detail line=5292 msg="vd-root received a packet(proto=1, 10.0.10.2:512->10.0.10.1:2048) from Spoke. type=8, code=0, id=512, seq=1." id=20085 trace_id=2 func=resolve_ip_tuple_fast line=5367 msg="Find an existing session, id-00000340, original direction" id=20085 trace_id=3 func=print_pkt_detail line=5292 msg="vd-root received a packet(proto=1, 10.0.10.2:512->10.0.10.1:2048) from Spoke. type=8, code=0, id=512, seq=2." id=20085 trace_id=3 func=resolve_ip_tuple_fast line=5367 msg="Find an existing session, id-00000340, original direction" id=20085 trace_id=4 func=print_pkt_detail line=5292 msg="vd-root received a packet(proto=1, 10.0.10.2:512->10.0.10.1:2048) from Spoke. type=8, code=0, id=512, seq=3." id=20085 trace_id=4 func=resolve_ip_tuple_fast line=5367 msg="Find an existing session, id-00000340, original direction" id=20085 trace_id=5 func=print_pkt_detail line=5292 msg="vd-root received a packet(proto=1, 10.0.10.2:512->10.0.10.1:2048) from Spoke. type=8, code=0, id=512, seq=4." id=20085 trace_id=5 func=resolve_ip_tuple_fast line=5367 msg="Find an existing session, id-00000340, original direction"

Could anyone please provide a fix or a way to further troubleshoot? 

[thanhletrung85]

Did you resolve your problem? I got exactly your issue. Please help me to solve it.

[ede_pfau]

While debugging, I would focus on policies and the routing protocol. Make sure that BGP is working (debug, cut links to provoke route changes etc.).

By using ADVPN you avoid to tunnel spoke-to-spoke traffic through the hub - is that what you need, and what you test? If this point is not so important for your use case you could focus on hub-and-spoke, supernetting all spoke networks etc., and would avoid having to debug the routing protocol.

Of course, it all depends on the scale.

[noother10]

The fix was to enable net-device on phase1 interface of the hub. Apparently we could also use tunnel-search nexthop, though I've not tested it, but it's supposed to direct traffic based on the next hop from routing protocols such as BGP. It can be set to selectors if you're not using wildcard (0.0.0.0/0.0.0.0) selectors.

[thanhletrung85]

My diagram as attach.

Hub config:

FG1 # show vpn ipsec phase1-interface config vpn ipsec phase1-interface     edit "ADVPN"         set type dynamic         set interface "port1"         set peertype any         set proposal des-sha1         set add-route disable         set auto-discovery-sender enable         set psksecret ENC QMvrwngyaW0WAtYN78oW3bjb2fT9yNON05BJA1JdoY2gp4vwj8mRr8xKc0rkQR6nHMGZlHyCJmnhEnooal+WTgp8eMs1aCUr1bgtZm0MQV45gUCezkbP5+9hCXW9i3KrQeCcCNrj9X3vdDsMT+Igm8kszN818rFEpKCG5kaLhEJnfPSjmeyjotQEaKMySMPlN2zbsw==     next end FG1 # show vpn ipsec phase2-interface config vpn ipsec phase2-interface     edit "ADVPN-P2"         set phase1name "ADVPN"         set proposal des-sha1     next end FG1 # show system interface ADVPN config system interface     edit "ADVPN"         set vdom "root"         set ip 192.168.168.1 255.255.255.255         set allowaccess ping         set type tunnel         set remote-ip 192.168.168.254 255.255.255.0         set snmp-index 12         set interface "port1"     next end FG1 # show firewall policy config firewall policy     edit 1         set name "To Spoke"         set uuid 395b68bc-401c-51e8-c909-fdd0933c73b3         set srcintf "port2"         set dstintf "ADVPN"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next     edit 2         set name "From spoke"         set uuid 2230163e-4156-51e8-c559-3c4833ac015f         set srcintf "ADVPN"         set dstintf "port2"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next     edit 3         set name "Spoke to spoke"         set uuid 3da4b3f2-4156-51e8-1f4d-9386759e92a0         set srcintf "ADVPN"         set dstintf "ADVPN"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next end FG1 # show router bgp config router bgp     set as 65000     set router-id 192.168.168.1     config neighbor-group         edit "ADVPN-PEERS"             set next-hop-self enable             set remote-as 65000             set route-reflector-client enable         next         edit "advpn_peers"             set remote-as 65000             set route-reflector-client enable         next     end     config neighbor-range         edit 1             set prefix 192.168.168.0 255.255.255.0             set neighbor-group "advpn_peers"         next     end     config network         edit 1             set prefix 172.16.1.0 255.255.255.0         next     end     config redistribute "connected"     end     config redistribute "rip"     end     config redistribute "ospf"     end     config redistribute "static"     end     config redistribute "isis"     end     config redistribute6 "connected"     end     config redistribute6 "rip"     end     config redistribute6 "ospf"     end     config redistribute6 "static"     end     config redistribute6 "isis"     end end FG1 config:

FG2 # show vpn ipsec phase1-interface config vpn ipsec phase1-interface     edit "ADVPN"         set interface "port1"         set peertype any         set proposal des-sha1         set add-route disable         set auto-discovery-receiver enable         set remote-gw 10.0.0.2         set psksecret ENC FNK4++NEvtWwUFjS/Bhp+Ydq2ksWTr3mLyymWI3TAAtZ5Z7G/Q2Hgd8FfagXVXGwt6xX8mYeeKEjfifObuO82VxahVst0fTkQY1oqNL6WFSmmDwpYnmAqxMIh25/yDXabmkXzP+H1e9TVditfyeYwFPt26lHpUmVmgn7KT9pVvhGXgmsskOQd91AvlGQ/2QzKpe14Q==     next end FG2 # show vpn ipsec phase2-interface config vpn ipsec phase2-interface     edit "ADVPN-P2"         set phase1name "ADVPN"         set proposal des-sha1     next end FG2 # show system interface ADVPN config system interface     edit "ADVPN"         set vdom "root"         set ip 192.168.168.2 255.255.255.255         set allowaccess ping         set type tunnel         set remote-ip 192.168.168.1 255.255.255.255         set snmp-index 12         set interface "port1"     next end FG2 # show firewall policy config firewall policy     edit 1         set name "To Hub"         set uuid 60ecb426-401c-51e8-24aa-89ea90a985e5         set srcintf "port2"         set dstintf "ADVPN"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next     edit 2         set name "From Hub"         set uuid 75551718-4158-51e8-57e5-aff965cea992         set srcintf "ADVPN"         set dstintf "port2"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next end FG2 # show router bgp config router bgp     set as 65000     set router-id 192.168.168.2     config neighbor         edit "192.168.168.1"             set next-hop-self enable             set soft-reconfiguration enable             set remote-as 65000         next     end     config network         edit 1             set prefix 172.16.2.0 255.255.255.0         next     end     config redistribute "connected"     end     config redistribute "rip"     end     config redistribute "ospf"     end     config redistribute "static"     end     config redistribute "isis"     end     config redistribute6 "connected"     end     config redistribute6 "rip"     end     config redistribute6 "ospf"     end     config redistribute6 "static"     end     config redistribute6 "isis"     end end

FG2 config:

FG3 # show vpn ipsec phase1-interface config vpn ipsec phase1-interface     edit "ADVPN"         set interface "port1"         set peertype any         set proposal des-sha1         set add-route disable         set auto-discovery-receiver enable         set remote-gw 10.0.0.2         set psksecret ENC YRsUl0Trdul0DnymG1mJQ+EwVUIIE4TQ2/XDTjEhpCs3vVghrom0DkrZQB2oJymrH9XgwrIvGTZxwD9lW4z9xgd5lEAHLF8zP3+DfL3MhWjyNURwaEKvIIuTUNFHU3BgnCRIbDfLNV3T3o/2qlijss7XyIDpdogx8pca74aX+ZGt+OVF8h5tFEG1zCx7cAX+3fwl2w==     next end FG3 # show vpn ipsec phase2-interface config vpn ipsec phase2-interface     edit "ADVPN-P2"         set phase1name "ADVPN"         set proposal des-sha1     next end FG3 # show system interface ADVPN config system interface     edit "ADVPN"         set vdom "root"         set ip 192.168.168.3 255.255.255.255         set allowaccess ping         set type tunnel         set remote-ip 192.168.168.1 255.255.255.255         set snmp-index 12         set interface "port1"     next end FG3 # show firewall policy config firewall policy     edit 1         set name "To Hub"         set uuid fb380bec-401d-51e8-f59e-63e73c3ab1b0         set srcintf "port2"         set dstintf "ADVPN"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next     edit 2         set name "From Hub"         set uuid 86b97ca8-415b-51e8-aea9-a5d88890a600         set srcintf "ADVPN"         set dstintf "port2"         set srcaddr "all"         set dstaddr "all"         set action accept         set schedule "always"         set service "ALL"     next end FG3 # show router bgp config router bgp     set as 65000     set router-id 192.168.168.3     config neighbor         edit "192.168.168.1"             set remote-as 65000         next     end     config network         edit 1             set prefix 172.16.3.0 255.255.255.0         next     end     config redistribute "connected"     end     config redistribute "rip"     end     config redistribute "ospf"     end     config redistribute "static"     end     config redistribute "isis"     end     config redistribute6 "connected"     end     config redistribute6 "rip"     end     config redistribute6 "ospf"     end     config redistribute6 "static"     end     config redistribute6 "isis"     end end All tunnel are up. But hub can only receive bgp from first connected spoke.

Please help me :)

[noother10]

Test to see if you can ping between the tunnel interfaces (192.168.168.x). My issue was that while the tunnel connected, only the first connected tunnel could actually communicate over it, the other tunnel wouldn't allow anything.

The fix was to:

config vpn ipsec phase1-interface

edit Hub

set net-device enable

end

[btp]

(Please mark the post as Answered if you feel it does so..)

In this case I can ping the IPSEC interface of the spokes from the hub. But the routes are not installed at the hub.

config vpn ipsec phase1-interface
    edit "PRIMARY"
        set type dynamic
        set interface "uplink1_lab"
        set peertype any
        set proposal aes256-sha256
        set add-route disable
        set dhgrp 5
        set auto-discovery-sender enable
        set net-device enable <<---
        set psksecret ENC ****
    next
end

HUB (VPN) # get router info bgp neighbors 10.254.0.2 received-routes
BGP table version is 6, local router ID is 172.16.3.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 10.41.1.0/24 10.254.0.2 0 65501 ?
*> 10.42.1.0/24 10.254.0.2 0 65501 ?
(...)

HUB (VPN) # get router info routing-table det 10.41.1.0/24
% Network not in table

The prefixes are not installed in the RIB:

HUB (VPN) # get router info kernel | grep 10.41.1

HUB (VPN) #

In this case, owever,  everything works fine. I can ping both IPSEC interface of the spokes from the hub, and I can ping through the tunnel.

config vpn ipsec phase1-interface

    edit "PRIMARY"
        set type dynamic
        set interface "uplink1_lab"
        set peertype any
        set proposal aes256-sha256
        set add-route disable
        set dhgrp 5
        set auto-discovery-sender enable
        set tunnel-search nexthop <<---
        set psksecret ENC ****
    next
end

HUB (VPN) # get router info rout all

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

B 10.41.1.0/24 [20/0] via 10.254.1.2, SECONDARY, 00:05:14
                     [20/0] via 10.254.0.2, PRIMARY, 00:05:14
B 10.41.3.0/24 [20/0] via 10.254.0.3, PRIMARY, 00:03:52
                     [20/0] via 10.254.1.3, SECONDARY, 00:03:52
C 10.60.0.0/24 is directly connected, CENTRAL-SERVER
C 10.254.0.0/24 is directly connected, PRIMARY
                      is directly connected, PRIMARY
C 10.254.1.0/24 is directly connected, SECONDARY
                      is directly connected, SECONDARY
       
HUB (VPN) # get router info kernel | grep 10.41.1
tab=254 vf=2 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->10.41.1.0/24 pref=0.0.0.0

I am then able to ping from one spoke to the other, through the hub (I have asymroute enabled);

id=20085 trace_id=55 func=print_pkt_detail line=5311 msg="vd-VPN received a packet(proto=1, 10.41.1.1:5888->10.41.3.1:2048) from SECONDARY. type=8, code=0, id=5888, seq=0."
id=20085 trace_id=55 func=init_ip_session_common line=5470 msg="allocate a new session-00000742"
id=20085 trace_id=55 func=vf_ip_route_input_common line=2576 msg="find a route: flag=00000000 gw-10.254.0.3 via PRIMARY"
id=20085 trace_id=55 func=fw_forward_handler line=743 msg="Allowed by Policy-3:"
id=20085 trace_id=55 func=ipsecdev_hard_start_xmit line=635 msg="enter IPsec interface-PRIMARY"
id=20085 trace_id=55 func=esp_output4 line=892 msg="IPsec encrypt/auth"
id=20085 trace_id=55 func=ipsec_output_finish line=527 msg="send to 172.16.54.1 via intf-uplink1_lab"