Hot!Virtual Server Cannot Connect to Outside

Author
polarpanda
New Member
  • Total Posts : 19
  • Scores: 0
  • Reward points: 0
  • Status: offline
2020/01/07 11:40:14 (permalink)
0

Virtual Server Cannot Connect to Outside

Hi there,
         I'm new to fortigate. I am trying to figure out why a virtual server stuck at firewall without denied policy setup. It used to work. When I did traceroute on the server, it stopped at the firewall. I don't see any policy to deny the server. Is there any other troubleshooting I can do? Thank you. 
#1

19 Replies Related Threads

    _aey_
    New Member
    • Total Posts : 12
    • Scores: 0
    • Reward points: 0
    • Joined: 2019/08/26 01:53:58
    • Location: Turkey
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 02:29:33 (permalink)
    0
    Hi,
     
    Can you check the logs ? When you write source and destination IP addresses in the logs filter, pls check the policy column and see the matched policy name. If the traffic match with deny policy, you should create a new policy to allow traffic.
    #2
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 02:34:18 (permalink)
    0
    hi,
     
    it works the other way around: without any ALLOWING policy there won't be any traffic. There's an implicit DENY ALL policy at the end of the policy table, invisible.
    If you do have an outbound policy, be sure to have NAT checked (to the WAN's interface address) or reply traffic won't make it back to the FGT.

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #3
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 07:46:56 (permalink)
    0
    engineer56
    Hi,
     
    Can you check the logs ? When you write source and destination IP addresses in the logs filter, pls check the policy column and see the matched policy name. If the traffic match with deny policy, you should create a new policy to allow traffic.





    Thank you for replying my post, aey. Yes, I checked the logs and found the policy. It's the group from inside to outside internet with accepted source ip "all" (0.0.0.0/0) to destination ip "all" (0.0.0.0/0).
    post edited by polarpanda - 2020/01/08 07:57:37
    #4
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 07:49:22 (permalink)
    0
    ede_pfau
    hi,
     
    it works the other way around: without any ALLOWING policy there won't be any traffic. There's an implicit DENY ALL policy at the end of the policy table, invisible.
    If you do have an outbound policy, be sure to have NAT checked (to the WAN's interface address) or reply traffic won't make it back to the FGT.


    Thank you for replying my post,ede. Yes, I knew that. So i have a question: does each server have to have its own policy in the firewall, even virtual server? If yes, I have two other vm servers in the same location (nutanix). I don't see both two servers have their own ALLOWING policy, but they're able to route outside internet.
     
    Ede, another amazing expert in this post told me to check logs and I did. The policy it's in is source all (0.0.0.0/0) to destination all (0.0.0.0/0)
    post edited by polarpanda - 2020/01/08 07:56:53
    #5
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 08:04:36 (permalink)
    0
    OK, so a 'all-to-all ACCEPT' policy is good for all servers/hosts on your LAN.
    If only this one server does not correctly connect to a host on the internet, you could look into the traffic using the CLI (console window):
    diag debug enable
    diag sniffer packet any 'host 192.168.456.789 and icmp' 4
     
    where you substitute the fake address with the source address of the server on your LAN. Then, start a ping on that server to 8.8.8.8 and record the output. You should see 'ICMP request' and 'ICMP reply' packets.
    Maybe you could copy&paste the output and post it here.

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #6
    emnoc
    Expert Member
    • Total Posts : 5509
    • Scores: 355
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 08:18:10 (permalink)
    0
    Agreed use diag debug flow, this ensures that the traffic that's expected is or is not reaching the firewall and the output will show allow/drop and any nat if applicable. It's always your 1st step in diagnostics , imho
     
    Ken Felix

    PCNSE 
    NSE 
    StrongSwan  
    #7
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 08:48:15 (permalink)
    0
    Thank you Ede. Before I did the debug you recommended, I did tracert via CMD on the server. It cannot even reach the firewall this time. So i think it might not be the issue.
     
    Do you think all to all would jam the traffic? If all the servers go through all to all, it would be really slow! So what's the best practice of policies for servers?
     
    Thank you!!
    #8
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 09:19:36 (permalink)
    0
    Let's first fix the problem you posted. I don't think (at all) that traffic jam prevents replies from coming in.
    If your server cannot reach the FGT check it's routing. The default route must point to the FGT's LAN interface address. A traceroute should at least show that ICMP traffic reaches the gateway.
     
    @emnoc: Happy New Year, Ken! I knew you'd agree to use 'diag debug flow', even if I suggested 'diag sniffer'... :-)
    and you're right!

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #9
    emnoc
    Expert Member
    • Total Posts : 5509
    • Scores: 355
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 09:24:35 (permalink)
    0
    Ede  no problem. My hunch was right that the traffic is not reaching the  FW. BTW I will be Berlin in mid-Feb, I will try to ping you when I'm in Frankfurt if time persist.
     
    Ken Felix

    PCNSE 
    NSE 
    StrongSwan  
    #10
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 09:46:26 (permalink)
    0
    Thank you! So i executed the diagnosis commands, then I did ping on the server. Nothing showed up on the CLI.  Then I did tracert 8.8.8.8, it did reach the firewall this time. So i'm confused why I don't see any ICMP capture.
     
     
    Firewall01 $ diag sniffer packet any 'host 10.x.x.x and icmp' 4 
    interfaces=[any]
    filters=[host 10.x.x.x and icmp]

    post edited by polarpanda - 2020/01/08 09:55:38
    #11
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 09:57:43 (permalink)
    0
    Did you
    1- 'diag debug enable'
    2- check the gateway setting on the server?

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #12
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 10:01:12 (permalink)
    0
    Sorry, it's my fault, ede. I was working on another similar issue, so I accidentally chose another firewall. Here is result, but I don't see the ICMP keywords you mentioned:
     
     
    5.126853 portxx in 10.x.x.x -> 8.8.8.8: icmp: echo request
    9.756878 portxx in 10.x.x.x -> 8.8.8.8: icmp: echo request
    14.757128 portxx in 10.x.x.x -> 8.8.8.8: icmp: echo request
    19.757406 portxx in 10.x.x.x -> 8.8.8.8: icmp: echo request

    #13
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 10:11:49 (permalink)
    0
    Hi Ken,
     
            Thank you for the advise. I'm really not familiar with fortinet commands. What's the different between your suggestion and Edes'?
    #14
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 10:24:00 (permalink)
    0
    OK, so the gateway setting on the server is correct, traffic to internet is hitting the FGT.
    Now you need to use the 'diag debug flow' command -

    diag debug flow filter clear
    diag debug flow filter proto 1
    diag debug flow show cons ena
    diag debug flow show fun ena
    diag debug flow show iprop ena
    diag debug flow trace start 10

     
    and ping away...trace will show you 10 events. Post one here.
    (don't worry if you receive errors with the 'show' commands, they depend on the FOS version)

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #15
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 10:40:40 (permalink)
    0
    diag debug flow show cons ena

    command not available
    #16
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 14:06:30 (permalink)
    0
    as I wrote, don't worry about errors...in v6.x, 'show console enable' is enabled by default, and not changeable anymore. Results?

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #17
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/08 14:28:40 (permalink)
    0
    ede_pfau
    as I wrote, don't worry about errors...in v6.x, 'show console enable' is enabled by default, and not changeable anymore. Results?




    Sorry, Ede. I understood now. Here is one of the events' result:
     
    2020-01-08 10:38:06 id=20085 trace_id=3016 func=print_pkt_detail line=5375 msg="vd-root received a packet(proto=1, 10.1.x.x:471xx->10.101.x.x:0) from local. type=0, code=0, id=471xx, seq=179."
    2020-01-08 10:38:06 id=20085 trace_id=3016 func=resolve_ip_tuple_fast line=5450 msg="Find an existing session, id-0a24f499, reply direction"
    #18
    ede_pfau
    Expert Member
    • Total Posts : 6186
    • Scores: 510
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/12 03:07:28 (permalink)
    0
    That is not sufficient to tell. Could you post more output, like 10 messages?

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #19
    polarpanda
    New Member
    • Total Posts : 19
    • Scores: 0
    • Reward points: 0
    • Status: offline
    Re: Virtual Server Cannot Connect to Outside 2020/01/13 09:50:42 (permalink)
    0
    Thank you for following it up, Ede. We found the problem. It's caused by our nutanix configuration. So the firewall site is all good. Thanks.
    #20
    Jump to:
    © 2020 APG vNext Commercial Version 5.5