Hot!Execute traceroute showing first and last hop 127.0.0.1 for connected subnet

Author
Deep Banerji
New Member
  • Total Posts : 6
  • Scores: 0
  • Reward points: 0
  • Joined: 2020/11/05 08:38:58
  • Status: offline
2020/11/05 08:52:07 (permalink)
0

Execute traceroute showing first and last hop 127.0.0.1 for connected subnet

I have a FortiGate 100e device in which I have taken out one LAN port and set WAN role on it. I have assigned a /30 subnet IP address to the port. The port is up and I can PING it from other zones. However, I cannot PING the remote IP address of the /30 subnet. Execute traceroute shows the only hop as 127.0.0.1. I have deployed ANY-ANY policy from LAN to the above interface but PING from LAN workstation to remote /30 IP address gets DESTINATION HOST UNREACHABLE reply from firewall. I am at my wit's end. Please help.
#1

12 Replies Related Threads

    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 00:29:13 (permalink)
    0
    is the remote IP address in the arp table?
     
    get sys arp
     
    is the IP 127.0.0.1 configured on the firewall?
    #2
    Deep Banerji
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2020/11/05 08:38:58
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 00:47:02 (permalink)
    0
    'get sys arp' is not showing the port in question. I ran 'diag sniff packet <port> 4' which is only showing arp requests. No arp replies. But when I connect the cable to a laptop it works. Btw, this is an Internet link with publicly available IP.  Is there some problem with FGT ARP request which makes the next-hop ignore the ARP request?
    No, 127.0.0.1 is not configured on the FGT. Firmware version is v5.6.4
    post edited by Deep Banerji - 2020/11/07 00:51:50
    #3
    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 04:40:31 (permalink)
    0
    Deep Banerji
    Is there some problem with FGT ARP request which makes the next-hop ignore the ARP request?



    there might be if the configuration isnt correct
     
    if you connect a laptop you say it works, you connect the laptop to the same interface on the internet modem / router?
     
    do you configure an IP on the laptop or use DHCP?
     
    what is the one LAN port configuration?
     
    can you share some of the arp request sniffer output?
     
    also please upgrade, 5.6 is not supported any more. not going to fix this, but just a good idea.
    #4
    Deep Banerji
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2020/11/05 08:38:58
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 06:36:45 (permalink)
    0
    Yes I'm connecting the same cable and assigning the same IP (x.y.z.189/30) manually on the FGT-100E and laptop. The LAN port members were port1 to port16 initially. I took out port15 and port16 to assign WAN role. 
     
    edit "lan"
    set vdom "root"
    set ip 10.14.50.1 255.255.255.0
    set allowaccess ping https ssh snmp http
    set type hard-switch
    set alias "LAN1"
    set device-identification enable
    set role lan
    set snmp-index 9
    next
     
    edit "port16"
    set vdom "root"
    set ip x.y.z.189 255.255.255.252
    set allowaccess ping
    set type physical
    set role wan
    set snmp-index 11
    next
     
    edit "wan1"
    set vdom "root"
    set ip 10.217.7.2 255.255.255.252
    set allowaccess ping https ssh http telnet
    set type physical
    set role wan
    set snmp-index 3
    next
     
    Next-hop IP is x.y.z.190/30 . I have tried commands like 'set arpforward enable' and 'set l2forward enable' to no avail. WAN1 and WAN2 ports are working fine.
     
    # diag sniff packet port16 none 4 6
    interfaces=[port16]
    filters=[none]
    0.620092 port16 -- truncated 802.3ad LACPDU 64
    1.644076 port16 -- truncated 802.3ad LACPDU 64
    2.668155 port16 -- truncated 802.3ad LACPDU 64
    3.692160 port16 -- truncated 802.3ad LACPDU 64
    4.716111 port16 -- truncated 802.3ad LACPDU 64
    5.740114 port16 -- truncated 802.3ad LACPDU 64
     
    # diag sniff packet any 'arp' 4
    interfaces=[any]
    filters=[arp]
    1.067289 port16 out arp who-has x.y.z.190 tell x.y.z.189
    1.617282 lan out arp who-has 10.14.50.81 tell 10.14.50.1
    1.617602 lan in arp reply 10.14.50.81 is-at 0:50:56:ae:af:cf
    2.067292 port16 out arp who-has x.y.z.190 tell x.y.z.189
    3.071994 port16 out arp who-has x.y.z.190 tell x.y.z.189
    3.905078 lan in arp who-has 10.11.50.10 tell 10.11.50.77
    4.067315 port16 out arp who-has x.y.z.190 tell x.y.z.189
    4.607669 lan in arp who-has 10.11.50.10 tell 10.11.50.77
    4.720917 lan in arp who-has 10.14.50.252 tell 10.14.50.252
    5.067294 port16 out arp who-has x.y.z.190 tell x.y.z.189
    5.607677 lan in arp who-has 10.11.50.10 tell 10.11.50.77
    6.073089 port16 out arp who-has x.y.z.190 tell x.y.z.189
    6.950751 lan in arp who-has 10.11.50.10 tell 10.11.50.21
    7.067287 port16 out arp who-has x.y.z.190 tell x.y.z.189
     
    I have another ISP link on port15 and it is having the same issue.
     
    #5
    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 07:27:14 (permalink)
    0
    you seem to be sending out or receiving LACP traffic on port16, is port16 part of a link aggregate? or is the ISP router doing LACP? please turn that off for now.
     
    also be sure you turn everything back to default which you have been changing, in the end it might not work because of all the non standard settings.
     
     
    #6
    Deep Banerji
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2020/11/05 08:38:58
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/07 09:10:22 (permalink)
    0
    No, I haven't configured port16 as member of any aggregate. I can't do anything about the ISP end config but given that it works on laptop it shouldn't be an issue.
    After testing various configurations I always revert to the default/standard state.
    I feel that somehow it has got to do with the fact that port16 was extracted from LAN group. That was the time when it was part of any aggregate. I read that FortiOS v5.4 onwards the internal-switch-mode is 'interface' by default so I just had to de-member it from LAN group. The device is suffering from some kind of 'LAN hangover' if I may put it that way. I have tried rebooting the device also. Nothing works. 
    #7
    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 00:53:14 (permalink)
    0
    Deep Banerji
    The device is suffering from some kind of 'LAN hangover' if I may put it that way. I have tried rebooting the device also. Nothing works. 

    that sounds very interesting but my experience is that things work or not, a port doesn't remember what they used to be. theoretically the change might seem executed but wasn't actually performed. a reboot should then fix that for sure.
     
    lets try to get the exact setup right because you talk about port16, but also mention port15, share the config on wan1 and you arp requests show lan in arp who-has 10.11.50.10 tell 10.11.50.77 which isnt the subnet for lan at all.
     
    how many interface does the ISP router / modem have? please only connect one form the FortiGate.
     
    are you using the copper or SFP part of port16? do you have a SFP inserted?
     
    keep it simple to start with and don't make changes to default configuration. just wondering are you able to factory reset this FortiGate and start over?
     
    parallel to that, please try this, connect port16 to the ISP router / modem.
     
    perform:
     
    show | grep -f port16
     
    show system interface port16
     
    show system virtual-switch
     
    show system switch-interface
     
    execute ping x.y.z.190
     
    get sys arp
     
    get router info routing all
     
    diagnose netlink aggregate list
     
    diagnose hardware deviceinfo nic port16
     
    diag sniff packet port16 '' 4 100 l
     
     
    post edited by boneyard - 2020/11/08 00:56:08
    #8
    Deep Banerji
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2020/11/05 08:38:58
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 03:22:07 (permalink)
    0
    10.11.50.0/24 is a known subnet and is connected to the LAN subnet through an L3 switch. I don't know how those broadcasts are reaching my LAN port.
     
    # show | grep -f port16
    config system interface
    edit "port16" <---
    set vdom "root"
    set ip x.y.z.189 255.255.255.252
    set allowaccess ping
    set type physical
    set role wan
    set snmp-index 11
    next
    end
    config firewall policy
    edit 5
    set name "test"
    set uuid <>
    set srcintf "lan"
    set dstintf "port16" <---
    set srcaddr "all"
    set dstaddr "all"
    set action accept
    set schedule "always"
    set service "ALL"
    set logtraffic all
    set nat enable
    next
    end
    config router static
    edit 6
    set gateway x.y.z.190
    set device "port16" <---
    next
    end
    # diag sniff packet port16 '' 4 100 l
    interfaces=[port16]
    filters=[]
    2020-11-08 16:11:45.771877 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:45.888782 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:46.776782 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:46.912853 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:47.771873 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:47.936822 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:48.771879 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:48.960825 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:49.776967 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:49.984818 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:50.771880 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:51.008839 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:51.771879 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:11:52.032835 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:11:52.776850 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    ^C
    17 packets received by filter
    0 packets dropped by kernel
    # show sys int port16
    config system interface
    edit "port16"
    set vdom "root"
    set ip x.y.z.189 255.255.255.252
    set allowaccess ping
    set type physical
    set role wan
    set snmp-index 11
    next
    end
    # show sys virtual-switch
    config system virtual-switch
    edit "lan"
    set physical-switch "sw0"
    config port
    edit "port1"
    next
    edit "port2"
    next
    edit "port3"
    next
    edit "port4"
    next
    edit "port5"
    next
    edit "port6"
    next
    edit "port7"
    next
    edit "port8"
    next
    edit "port9"
    next
    edit "port10"
    next
    edit "port11"
    next
    edit "port12"
    next
    edit "port13"
    next
    edit "port14"
    next
    end
    next
    end
    # show sys switch-interface
    config system switch-interface
    end
    # show sys switch-interface
    config system switch-interface
    end
    # execute ping x.y.z.190
    PING x.y.z.190 (x.y.z.190): 56 data bytes
    --- x.y.z.190 ping statistics ---
    5 packets transmitted, 0 packets received, 100% packet loss

    # get sys arp
    Address Age(min) Hardware Addr Interface
    10.14.50.3 0 80:30:e0:84:dc:00 lan
    10.14.50.80 0 00:0c:29:2c:10:30 lan
    10.14.50.99 0 f4:03:43:b9:cd:a0 lan
    10.217.7.5 0 00:03:0f:12:e0:3a wan2
    10.14.50.79 0 00:0c:29:d2:45:39 lan
    10.217.7.1 0 00:03:0f:12:e0:3a wan1
    10.14.50.231 0 c4:00:ad:0b:07:33 lan
    10.14.50.81 0 00:50:56:ae:af:cf lan
    # get router info routing all
    Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
    O - OSPF, IA - OSPF inter area
    N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
    E1 - OSPF external type 1, E2 - OSPF external type 2
    i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
    * - candidate default
    S* 0.0.0.0/0 [10/0] via x.y.z.190, port16
    S 10.11.0.0/16 [10/0] via 10.14.50.3, lan
    S 10.12.0.0/16 [10/0] via 10.14.50.3, lan
    C 10.14.50.0/24 is directly connected, lan
    S 10.217.0.0/16 [10/0] via 10.217.7.1, wan1
    [10/0] via 10.217.7.5, wan2
    C 10.217.7.0/30 is directly connected, wan1
    C 10.217.7.4/30 is directly connected, wan2
    C x.y.z.188/30 is directly connected, port16
    C a.b.c.112/29 is directly connected, port15

    # diag netlink aggregate list
    List of 802.3ad link aggregation interfaces:
    # diag netlink aggregate list
    List of 802.3ad link aggregation interfaces:
     
    # diag hardware deviceinfo nic port16
    Description :FortiASIC NP6LITE Adapter
    Driver Name :FortiASIC NP6LITE Driver
    Board :100E
    lif id :21
    lif oid :85
    netdev oid :85
    Current_HWaddr e8:1c:ba:07:0e:ab
    Permanent_HWaddr e8:1c:ba:07:0e:ab
    ========== Link Status ==========
    Admin :up
    netdev status :up
    autonego_setting:1
    link_setting :1
    speed_setting :10
    duplex_setting :0
    Speed :1000
    Duplex :Full
    link_status :Up
    ============ Counters ===========
    Rx Pkts :0
    Rx Bytes :0
    Tx Pkts :100724
    Tx Bytes :4230454
    Host Rx Pkts :163318
    Host Rx Bytes :10452352
    Host Tx Pkts :100725
    Host Tx Bytes :4230496
    Host Tx dropped :0

    # diag sniffer packet port16 '' 4 100 l
    interfaces=[port16]
    filters=[]
    2020-11-08 16:25:53.522050 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:53.765784 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:54.527077 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:54.789781 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:55.522045 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:55.813793 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:56.522045 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:57.527064 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:57.861866 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:58.522047 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:58.885802 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:59.522043 port16 -- arp who-has x.y.z.190 tell x.y.z.189
    2020-11-08 16:25:59.909816 port16 -- truncated 802.3ad LACPDU 64
    ^C
    15 packets received by filter
    0 packets dropped by kernel
    #9
    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 03:58:53 (permalink)
    0
    thanks for all the output, i don't see anything what looks wrong except these:
     
    2020-11-08 16:25:55.813793 port16 -- truncated 802.3ad LACPDU 64
    2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64
     
    2020-11-08 16:25:56.837797 port16 -- truncated 802.3ad LACPDU 64
     
    you should not see LACPDU on a non link aggregate interface. the fact that it works on the laptop surprises me but perhaps something is configured for it or it auto reacts in a way the ISP router / modem likes better.
     
    the ISP router / modem doesn't have another interface to use?
     
    can you request the ISP to provide an interface on the device without LACP?
     
    if you want to and have the time you could create a link aggregate on the FortiGate, make port16 a member of it and see if that gets a working situation. that does require removing the IP from port16, removing the firewall policy and the route. then putting them back on the link aggregate.
    #10
    Deep Banerji
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2020/11/05 08:38:58
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 05:00:28 (permalink)
    0
    boneyard 
    if you want to and have the time you could create a link aggregate on the FortiGate, make port16 a member of it and see if that gets a working situation. that does require removing the IP from port16, removing the firewall policy and the route. then putting them back on the link aggregate.



    Tried this. Didn't work. :(
    #11
    boneyard
    Gold Member
    • Total Posts : 364
    • Scores: 16
    • Reward points: 0
    • Joined: 2014/07/30 11:15:18
    • Status: offline
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 05:30:55 (permalink)
    0
    what does the diagnose sniffer packet for the link aggregate look like?
    #12
    emnoc
    Expert Member
    • Total Posts : 5860
    • Scores: 387
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: online
    Re: Execute traceroute showing first and last hop 127.0.0.1 for connected subnet 2020/11/08 21:08:06 (permalink)
    0
    This happens in all of my FGT that I manage. What I've notice, if the trace route is done to a "wan" or "port" interface that is not part of a virtual-switch it looks normal. If you do a trace route to a address connected to a port of a virtual-switch,  the 127.0.0.1 comes up
     
    e.g 
     
    MANHATTANSOUTH # diag ip arp list | grep wan
    index=8 ifname=wan2 xxx.xxx.1 00:1b:bc:11:43:1a state=00000004 use=61 confirm=47 update=27 ref=51
     
    MANHATTANSOUTH # execute traceroute xxx.xxx.200.1
    traceroute to xxx.xxx.200.1 (xxx.xxx.200.1), 32 hops max, 3 probe packets per hop, 72 byte packets
    1 xxx.xxx.200.1 0.373 ms 0.330 ms 0.173 ms
     
    and here's a LAN ( virtual-switch )
     
    MANHATTANSOUTH # execute traceroute 10.1.1.50
    traceroute to 10.1.1.50 (10.1.1.50), 32 hops max, 3 probe packets per hop, 72 byte packets
    1 127.0.0.1 <gearssdk.opswat.com> 2994.351 ms !H 2999.669 ms !H 2999.987 ms !H
     
    Opswat does end-point protection, so it's something in fortOS that using some protection. Fortinet is a partner of opswat. 
    reference
     
    https://www.opswat.com/partners/fortinet
     
    So if their is not problem with the connected host, I would chalk this up as cosmetic.
     
     
    Just my observations.
     
    Ken Felix

    PCNSE 
    NSE 
    StrongSwan  
    #13
    Jump to:
    © 2020 APG vNext Commercial Version 5.5